Failure pushing multi-arch images to Private AWS ECR with tag immutability enabled

[Vlaaaaaaad]

Hi,

As per the instructions in TROUBLESHOOTING.md for the docker/build-push-action GitHub Action, I am opening an issue here. If this is not the right place, please let me know. This could be an issue with the ECR implementation of manifest lists.

TL;DR, from Twitter:

Pushing a multi-arch (AMD, ARM) container image to ECR:

ECR Public: everything works 🎉
ECR Private, no tag immutability: everything works ✨

ECR Private, tag immutability enabled: docker/build-push-action fails, containerd works 🙃

---

Pushing a multi-arch (AMD, ARM) image to a private AWS ECR repository fails with unexpected status: 400 Bad Request. Pushing the same image to ECR, using containerd works.

I created an example repository to show the issue and the relevant config/logs. You can see the GitHub Action failing here and you can see containerd working here.

Relevant logs for the failure:

2021-03-01T13:06:04.8482250Z #21 exporting to image
2021-03-01T13:06:04.8483337Z #21 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
2021-03-01T13:06:07.6660224Z #21 pushing layers 3.3s done
2021-03-01T13:06:07.6663629Z #21 pushing manifest for ***.dkr.ecr.***.amazonaws.com/bug-report:buildkit-dfa6ddf4705485840c392c397106932c03bf4961
2021-03-01T13:06:08.4085934Z #21 pushing manifest for ***.dkr.ecr.***.amazonaws.com/bug-report:buildkit-dfa6ddf4705485840c392c397106932c03bf4961 0.7s done
2021-03-01T13:06:08.4088745Z #21 ERROR: failed commit on ref "manifest-sha256:672d30d763ba54a9c7f5169573b554f16efd3e8a25deb60413d259a29b204d76": unexpected status: 400 Bad Request
2021-03-01T13:06:08.4090060Z ------
2021-03-01T13:06:08.4090863Z  > exporting to image:
2021-03-01T13:06:08.4091431Z ------
2021-03-01T13:06:08.4093157Z error: failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:672d30d763ba54a9c7f5169573b554f16efd3e8a25deb60413d259a29b204d76": unexpected status: 400 Bad Request
2021-03-01T13:06:08.4160997Z ##[error]buildx call failed with: error: failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:672d30d763ba54a9c7f5169573b554f16efd3e8a25deb60413d259a29b204d76": unexpected status: 400 Bad Request

Funnily enough, if I go to the ECR repository and list the images, there is something pushed for both:

containerd-dfa6ddf4705485840c392c397106932c03bf4961   01 Mar 2021 15:06:51   380.83MB   sha256:d14dffbdc33c894279a22d6ff71671ab6d243a3b488563ad3360bdca781914a9
<untagged>                                            01 Mar 2021 15:06:50   370.38MB   sha256:2fd9e0a11c41f27986d6c0bc6b949a76b85bb9ade27685c4200b87f1c84e991b
<untagged>                                            01 Mar 2021 15:06:50   380.83MB   sha256:43c4984a930ced6b904820bcce54a85a31235cbd915ba3eaf45c31cc7810cf2b
buildkit-dfa6ddf4705485840c392c397106932c03bf4961     01 Mar 2021 15:06:08   380.83MB   sha256:2eb8f205617af17c0af81e1871b43f2d69a59b3c723a9c5bf61258b1712b9411

[crazy-max]

@Vlaaaaaaad Can you try the following permutations for the buildx step please:

• driver-opts: image=moby/buildkit:v0.8.1
• driver-opts: image=moby/buildkit:buildx-stable-1 (default)

[Vlaaaaaaad]

@crazy-max sure!

Here's v0.8.1: https://github.com/Vlaaaaaaad/ecr-dual-arch-immutable-buildkit-bug/pull/2/checks?check_run_id=2006372648
Here's buildx-stable-1: https://github.com/Vlaaaaaaad/ecr-dual-arch-immutable-buildkit-bug/pull/1/checks?check_run_id=2006373694

Unfortunately, the same error is reported. Let me know if the changes are not in the right place and I can re-test 🙂

[crazy-max]

@Vlaaaaaaad Ok thanks for your feedback. There is something odd and I think it's a scope issue on ECR. We have e2e tests with several registries to detect if there is something wrong early. And everything looks fine atm with our private ECR. Can you check if the the requirement to push to ECR with AmazonEC2ContainerRegistryPowerUser managed policy is ok? Thanks.

EDIT: Nevermind works fine with containerd. Must be something else.

[crazy-max]

@Vlaaaaaaad Wonder if this is not linked to a specific region. Can you disclose it? WDYT @jtoberon?

[Vlaaaaaaad]

@crazy-max hm... does your test ECR repo have "Immutable tags" enabled? I could not find a definition for the repo itself. The docker/build-push-action action works great on non-immutable repos.

The repo is in eu-west-1.
I doubt there's a policy issue as the the IAM user used has both AmazonEC2ContainerRegistryPowerUser and the following policy attached:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:CompleteLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:InitiateLayerUpload",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage"
            ],
            "Resource": "arn:aws:ecr:eu-west-1:000000000000:repository/bug-report"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

I can test another region if you think it would help!

[jtoberon]

@omieomye ^^^

[tonistiigi]

I'm surprised you see it working in containerd. This has been a known error of containerd pusher that it repushes the tag multiple times. Maybe something has changed or containerd case is not quite the same.

[tonistiigi]

Looking at the code I think containerd does a trick on pushing by tag and digest together. In that case tags are ignored for objects where digest does not match. I think we should be able to do the same thing.