Cannot push docker image to Google Artifact Registry

[jacek-jablonski]

Hi,

I've got quite a simple workflow using build-push-action v2, but I am unfortunately unable to push image successfully to Google Artifact Registry.

Here is the workflow:

name: CI

on:
  push:
    tags:
      - 'v*.*.*'

env:
  REGISTRY: europe-west4-docker.pkg.dev
  PROJECT_ID: xxx
  REPOSITORY_ID: appconfig

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Prepare
        id: prepare
        run: |
          DOCKER_IMAGE="${REGISTRY}/${PROJECT_ID}/${REPOSITORY_ID}"
          VERSION=${GITHUB_REF#refs/tags/}
          TAGS="${DOCKER_IMAGE}:${VERSION}"
          if [[ $VERSION =~ ^v[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
            MINOR=${VERSION%.*}
            MAJOR=${MINOR%.*}
            TAGS="$TAGS,${DOCKER_IMAGE}:${MINOR},${DOCKER_IMAGE}:${MAJOR},${DOCKER_IMAGE}:latest"
          elif [ "${{ github.event_name }}" = "push" ]; then
            TAGS="$TAGS,${DOCKER_IMAGE}:sha-${GITHUB_SHA::8}"
          fi
          echo ::set-output name=version::${VERSION}
          echo ::set-output name=tags::${TAGS}
          echo ::set-output name=created::$(date -u +'%Y-%m-%dT%H:%M:%SZ')
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          buildkitd-flags: --debug

      - name: Login to GCR
        uses: docker/login-action@v1
        with:
          registry: ${{ env.REGISTRY }}
          username: _json_key
          password: ${{ secrets.GCP_SA_KEY }}

      - name: Build and push
        id: docker-build
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          target: bin
          push: true
          tags: ${{ steps.prepare.outputs.tags }}

It is failing with:

#10 pushing layers
#10 pushing layers 3.2s done
#10 pushing manifest for europe-west4-docker.pkg.dev/xxx/appconfig:v0.1.0
#10 pushing manifest for europe-west4-docker.pkg.dev/xxx/appconfig:v0.1.0 0.4s done
#10 ERROR: failed commit on ref "manifest-sha256:39c07bc2a80624b0ae6bb3c7a616b31a4ea846f8d679aca8835702328c57dccb": unexpected status: 400 Bad Request
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:39c07bc2a80624b0ae6bb3c7a616b31a4ea846f8d679aca8835702328c57dccb": unexpected status: 400 Bad Request

I tried to debug it using a troubleshooting note, but it seems that ctr accepts only docker login and password, but not GCP's service account JSON file.

Here is a full log of workflow:
1_docker.txt.zip

[crazy-max]

@jacek-jablonski

Looks like the same issue as docker/setup-buildx-action#29. Can you try my suggestion?

cc. @tonistiigi

[jacek-jablonski]

@crazy-max it worked! Thank you!

[crazy-max]

@jacek-jablonski

it seems that ctr accepts only docker login and password, but not GCP's service account JSON file.

You tried with:?

sudo ctr i push --user "_json_key:${{ secrets.GCP_SA_KEY }}" ${REGISTRY}/${PROJECT_ID}/${REPOSITORY_ID}:latest

[jacek-jablonski]

Yes, exactly. My GCP_SA_KEY is not base64 encoded.

[jacek-jablonski]

However above fix does not work for Google's Artifact Registry: europe-west4-docker.pkg.dev, where the problem still persists.

[crazy-max]

@jacek-jablonski Looks similar too oras-project/oras#157 (comment). You should ask to Google support about that.

[crazy-max]

@jacek-jablonski Will be solved through moby/buildkit#1730

[jacek-jablonski]

@crazy-max thanks for information!

[crazy-max]

@jacek-jablonski PR has been merged to master on buildkit repo. Can you test with the following step please? Thanks.

     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1
       with:
         driver-opts: image=moby/buildkit:master

[jacek-jablonski]

@crazy-max, unfortunately, it still does not work:

#12 pushing manifest for europe-west4-docker.pkg.dev/xxx/node:14.13.1-2 0.6s done
#12 ERROR: failed commit on ref "manifest-sha256:410534d3c19a2d7560776a25d7168786bb697dd345e60a101f83f83e15abedb1": unexpected status: 400 Bad Request
------
> exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:410534d3c19a2d7560776a25d7168786bb697dd345e60a101f83f83e15abedb1": unexpected status: 400 Bad Request
Error: buildx call failed with: failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:410534d3c19a2d7560776a25d7168786bb697dd345e60a101f83f83e15abedb1": unexpected status: 400 Bad Request

[jonjohnsonjr]

@jacek-jablonski I believe you need a third path component:

europe-west4-docker.pkg.dev/xxx/node:14.13.1-2

e.g. europe-west4-docker.pkg.dev/xxx/something-else/node:14.13.1-2

[tonistiigi]

I could not repro anymore with moby/buildkit:master moby/buildkit#1730 so need a new report with new reproduction details.

I was testing with gcr.io/<project>/<repo> if that makes any difference.

[tonistiigi]

I tested the *.pkg.dev host as well and indeed initially got unexpected status: 400 Bad Request.

In the logs:

time="2020-10-22T16:31:37Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"NAME_INVALID\",\"message\":\"Missing image name. Pushes should be of the form docker push HOST-NAME/PROJECT-ID/REPOSITORY/IMAGE\"}]}\n"

So what @jonjohnsonjr suggested is correct, you need third component in the name. After that everything worked fine again.

@jonjohnsonjr Could you look at updating containerd client code https://github.com/containerd/containerd/tree/master/remotes/docker so that it returns a more meaningful error instead of just 400 ?

[jonjohnsonjr]

It looks like containerd knows how to parse these errors, so I'm not sure why the response body is getting dropped.

I see that this error is passed through to the pipereader CloseWithError: https://github.com/containerd/containerd/blob/c408aa90867eb11bd444152a33b1a13da0e5b8f9/remotes/docker/pusher.go#L252

So I would expect that we'd read the error when we go to close the pipe: https://github.com/containerd/containerd/blob/c408aa90867eb11bd444152a33b1a13da0e5b8f9/remotes/docker/pusher.go#L335-L337

But instead we get down to here and see the unexpected status error, which also drops the body: https://github.com/containerd/containerd/blob/c408aa90867eb11bd444152a33b1a13da0e5b8f9/remotes/docker/pusher.go#L351

My guess is that you are racing the goroutine and you need to move this blocking channel read up above the pipe close to allow the goroutine a chance to finish calling CloseWithError before Commit calls Close to check for that error: https://github.com/containerd/containerd/blob/c408aa90867eb11bd444152a33b1a13da0e5b8f9/remotes/docker/pusher.go#L341

It would also feel correct to just throw an extra err := remoteserrors.NewUnexpectedStatusErr(resp) into pushWriter.Commit if debugging races sounds not fun.

Nevermind, looks like this isn't displayed. I assume there's something in containerd like "attempt to parse this response's body as a registry error, otherwise return a reasonable error" somewhere? This seems close.

I don't often (ever?) contribute to containerd so I don't have a lot of context here to confidently make this change (though I'm happy to) -- based on git blame: @dmcgowan does my assessment seem reasonable?

[crazy-max]

@jacek-jablonski You should now be able to use the default values for setup-buildx-action as the default buildkit container image has been updated (minus the required change in #171 (comment)):

     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1

[jacek-jablonski]

@crazy-max I can confirm that it was due to missing repository_id from the path. After adding a new segment, everything works as expected.

[jacek-jablonski]

Thanks everyone for your help.