aws_ecr_assets.DockerImageAsset: cannot deploy more than one stack simultaneously because of the image tag

[ukrainiansteak]

Describe the bug

In my aws-cdk stack deployed in eu-north-1 region, I define DockerImageAsset as follows:

        asset = DockerImageAsset(
            self,
            f"{config.stack_name}-docker-image",
            asset_name=f"{config.stack_name}-docker-image",
            directory=os.path.join("infrastructure", "dockerimage"),
        )

The asset is then referenced as an image for a CodeBuild project later in the code.

The problem arises when I deploy two or more stacks using the same template (obviously, the stack names are different) simultaneously. Image tag is most likely generated using the datetime info, which leads to the tags being the same for several images generated in the same timeframe (the same minute?).

Here's the error I receive when the stack fails:

4736 | stack-name-dev:  fail: 
docker push 528688307418.dkr.ecr.eu-north-1.amazonaws.com/cdk-hnb659fds-container-assets-account-id-eu-north-1:51687ba0f9469a37e6d0d900166b328d730f9c58751a61e8ecc7d25ec7dc3640 
exited with error code 1: tag invalid: The image tag '51687ba0f9469a37e6d0d900166b328d730f9c58751a61e8ecc7d25ec7dc3640' 
already exists in the 'cdk-hnb659fds-container-assets-account-id-eu-north-1' repository 
and cannot be overwritten because the repository is immutable.

If I wait at least one minute before I start another deploy, it all works fine. However, it is not realistic in the production environment, where several deploys can be started on the same account in the same minute.

The only customizable thing I see is the asset_name that doesn't seem to influence anything.

Expected Behavior

I expected several stacks to deploy fine and several images to be created with different tags.

Current Behavior

4736 | stack-name-dev:  fail: 
docker push 528688307418.dkr.ecr.eu-north-1.amazonaws.com/cdk-hnb659fds-container-assets-account-id-eu-north-1:51687ba0f9469a37e6d0d900166b328d730f9c58751a61e8ecc7d25ec7dc3640 
exited with error code 1: tag invalid: The image tag '51687ba0f9469a37e6d0d900166b328d730f9c58751a61e8ecc7d25ec7dc3640' 
already exists in the 'cdk-hnb659fds-container-assets-account-id-eu-north-1' repository 
and cannot be overwritten because the repository is immutable.

Reproduction Steps

Create a cdk template with a DockerImageAsset:

        asset = DockerImageAsset(
            self,
            f"{config.stack_name}-docker-image",
            asset_name=f"{config.stack_name}-docker-image",
            directory=os.path.join("infrastructure", "dockerimage"),
        )

Deploy several stacks using the template at the same time (same minute).

Possible Solution

Image tag is probably generated based on the time. Change the generation method.

Additional Information/Context

The cdk deploy command is run inside a Docker container (a separate build for every stack).

CDK CLI Version

2.133.0

Framework Version

No response

Node.js Version

18

OS

ubuntu:20.04

Language

Python

Language Version

Python 3.9.19

Other information

No response

[pahud]

I'd like to know more about how you deploy them. Are you deploying multiple stacks using the same DockerImageAsset in the same CDK app/stack for the same region using npx cdk deploy --all ?

[ukrainiansteak]

No, what I meant by simultaneous deployment is that we have a UI system where the user can click "Update" on several applications=stacks (any that have an update available). These stacks are deployed using the same cdk template, but the environment variables are different for each of them (e.g., config.stack_name from my code snippet).

So the region is the same but the environment variables used in the stacks are obviously different.

You can think of it in the following way:

We have stack A with the config.stack_name variable set to stack-a. So the DockerImageAsset code will technically be:

        asset = DockerImageAsset(
            self,
            f"stack-a-docker-image",
            asset_name=f"stack-a-docker-image",
            directory=os.path.join("infrastructure", "dockerimage"),
        )

And if we have stack B with the stack-b name, then this is the code that gets synthed into a cloudformation template:

        asset = DockerImageAsset(
            self,
            f"stack-b-docker-image",
            asset_name=f"stack-b-docker-image",
            directory=os.path.join("infrastructure", "dockerimage"),
        )

When the user clicks "Update", we start a build inside of a CodeBuild project. So for several "updates", several builds of the same CodeBuild project are run. So deploy of each stack is isolated, has its own environment variables and happens at the user click.

[pahud]

OK this sounds like a race condition when two isolate builds doing cdk deploy at the same time. Both bundle up the same docker image assets with the same tag and push at the same time.

According to this

aws-cdk/packages/aws-cdk-lib/aws-ecr-assets/lib/image-asset.ts

Line 364 in 8c39e81

* The tag of this asset when it is uploaded to ECR. The tag may differ from the assetHash if a stack synthesizer adds a dockerTagPrefix.

I guess you will need to define dockerTagPrefix in your synthesizer and generate unique prefix for it to avoid that. Let me know if it works for you.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.