support building docker images using BuildKit

[helmlover]

The claim "Docker [...] Works out of the box." is no longer true.
That is because docker now uses BuildKit per default, while the BuildKit Dockerfile syntax is not supported when building docker images with testcontainers-node (e.g. with GenericContainer.fromDockerfile(buildContext).build())

When building a Dockerfile containing BuildKit-features e.g.

RUN--mount=type=cache,id=maven,target=/root/.m2/repository mvn --batch-mode --no-transfer-progress dependency:resolve dependency:resolve-plugins

through testcontains-node (with export 'DEBUG=testcontainers*'), the output currently looks like:

2023-05-12T08:52:02.265Z testcontainers:build [localhost/43420fa7dc4a:2c9e53cede2c] {"errorDetail":{"message":"the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled"},"error":"the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled"}

Exporting export DOCKER_BUILDKIT=1 does not change the problem/output.

PS: Sibling issue in testcontainers-java: testcontainers/testcontainers-java#2857

[cristianrgreco]

Hi @helmlover, see the parent issue: docker/for-linux#1136. BuildKit does not yet seem supported over the Docker HTTP API, as such is only currently available via the CLI.

[osa0805]

Hi,
Is there a plan to add this support ?

[praveensvsrk]

+1, we need support for this....curious if there a workaround available to achieve the RUN cache like functionality without buildkit features?

[silh]

It looks like what's necessary is a support for creating a session and running it in dockerode. Go implementation for that in terraform was added here - https://github.com/kreuzwerker/terraform-provider-docker/pull/387/files#diff-4596d40531ae2e21f6074d104e6dc7317537946b56d95df847c9209dfbe30fceR329 The session run code is here

[mikeseese]

Note that the API does support using BuildKit with the version option: this issue has fixed in the correct repo: https://github.com/moby/moby/blob/master/api/swagger.yaml#L8722-L8731

The linked upstream issue (docker/for-linux#1136) is on a deprecated/seemingly abandoned repo.

[silh]

The main problem is with setting up a websocket connect for the session which is required for version 2. IIRC, dockerode (or its underling library docker-modem) didn't support that.

[mikeseese]

I'm not sure about that, or the requirements for this module (as I'm just bubbling up the finding as I saw others waiting on an issue on an abandoned repo), but I am able to use dockerode's buildImage with { version: "2" } as an option and BuildKit is used.

[schummar]

I don't think that's right. Passing { version: "2" } to dockerode's buildImage doesn't do anything. The build still fails when using some BuildKit dependent feature like --mount=type=cache.

I quickly looked into it at some point and my impression was that is really not straight forward using build version 2 on the docker api. I am not sure about any of this but my impression was: You to have to implement a gRPC server on your side, then hijack the http connection to start a session which allow the docker daemon to make calls on your end!? It's baffling api design if you ask me 😁

[mikeseese]

@schummar I guess my verification was that prior to adding { version: "2" } to the build options object, I would receive the error during building my image:

the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled

which referenced a COPY --chmod=755 ... line in my Dockerfile

and after adding it, I was able to build the image successfully 🤷 I can close my DefinitelyTyped PR if adding it doesn't support all BuildKit features

[silh]

@mikeseese that's still a valid option and should be added there. It exists in docker's API description - https://docs.docker.com/engine/api/v1.45/#tag/Image/operation/ImageBuild

[schummar]

@schummar I guess my verification was that prior to adding { version: "2" } to the build options object, I would receive the error during building my image:

the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled

which referenced a COPY --chmod=755 ... line in my Dockerfile

and after adding it, I was able to build the image successfully 🤷 I can close my DefinitelyTyped PR if adding it doesn't support all BuildKit features

Oh, I think you are right! When I was experimenting some weeks back I could not get this to work. But now I have tried it again and it seems it does work! Maybe something a new docker version quietly improved? Or I have just been doing it wrong all that time 🤣
That't good news, thanks!

[mikeseese]

Phew! I just finished creating a quick reproduction repo for quick testing; I was about to test --mount=type=cache, but I'll hold off since you verified yourself 👍 here's the repo in case it's helpful: https://github.com/mikeseese/dockerode-buildkit

[cristianrgreco]

Thank you @mikeseese for sharing the findings here! I don't know if I would've ever found that upstream issue 😄

[silh]

I was testing it locally as a test from my PR to support in testcontainers-go was passing for me too with current docker. However, when I was switching to the older docker it was failing. I enabled debug to understand why the problem happens, which lead me to this logs:

time="2024-06-11T19:17:05.199031371Z" level=debug msg=resolving host=registry-1.docker.io
time="2024-06-11T19:17:05.199064621Z" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/0.0.0+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
time="2024-06-11T19:17:07.605771997Z" level=debug msg="fetch response received" host=registry-1.docker.io response.header.content-length=157 response.header.content-type=application/json response.header.date="Tue, 11 Jun 2024 19:17:09 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=80.56.164.134 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/library/alpine/manifests/latest"
time="2024-06-11T19:17:07.605922831Z" level=debug msg=Unauthorized header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:library/alpine:pull\"" host=registry-1.docker.io
time="2024-06-11T19:17:07.606130331Z" level=info msg="trying next host" error="no active sessions" host=registry-1.docker.io

This happens when the base image is not present locally before the build, buildx will try to download it, but won't have auth data - it'll check it even for dockerhub - and in order to get that auth data it will try to find a session (somewhere inside github.com/containerd/containerd/remotes/docker/resolver.go retryRequest -> util/resolver/authorizer.go (dockerAuthorizer.AddResponses) -> sessionauth.GetTokenAuthority -> sessionManager.Any (should return any session).

After that I returned to the new docker version, wiped all local images and got the same error as with the old version.

Then I've decided to check the repo with an example provided by @mikeseese (https://github.com/mikeseese/dockerode-buildkit) and it the example there is a code to download a base image before building. When I commented out the pull code, deleted the pulled image and tried to run it I got the same no active session problem again.

$ node index.js
Building ./Dockerfile...
ERROR: alpine: no active sessions
ERROR: Failed to build image

[mikeseese]

Ya it seems like the { version: "2"} isn't going to get you full BuildKit support, but it can work in some limited scenarios. There's some discussion on apocas/dockerode#601 (comment) about adding the gRPC server implementation (here's where you have conflicting results @schummar; in one test scenario you likely didn't have the image pulled vs the other)

Long story short, I think it's safe to say that moby/the docker engine has support for this issue (for some minimum version of Docker), but each client will need to implement the BuildKit client/server to fully realize support, making this issue not blocked by an upstream issue.

[schummar]

Yeah, that's what I also found in #761. And that's also what stopped all my past experiments, because due to a bug (#771) pull: 'true' is currently always sent in testcontainers.
Since we are talking about huge limitations, it's up to the maintainers whether they want to include support right away (with docs discussing the limitations) or wait until dockerode supports it properly.

[schummar]

Made some progress: apocas/dockerode#766
But still a few things to sort out.