support for DOCKER_BUILDKIT=1 environment variable

[dominikzalewski]

Description

I would like to take advantage in my build of DOCKER BUILDKIT. Not only it runs faster, but has significant improvements like FROM ... AS that makes the images smaller.

I checked and it does not look like it is currently possible with the plugin.

I would like the plugin to respect the environment variable DOCKER_BUILDKIT, or have any other way of passing that information to the build engine.

[rhuss]

I don't know what DOCKER_BUILDKIT actually triggers on the client-side for changing the REST communication with the Docker daemon. d-m-p uses this REST API exclusively for talking to the Docker daemon, so it would be interesting what kind of HTTP query options are added to the build endpoint.

However, multi-stage should be already possible with dmp out of the box.

[famod]

Just ran into this, COPY --chmod=... is only available with BuildKit.

[famod]

Btw, I think there should (also) be a plugin config flag to enable BuilKit.

[rohanKanojia]

@famod : Do you if there is some kind of query parameter in REST API to honor DOCKER_BUILDKIT? Or is this something that is specific to docker CLI?

[famod]

@rohanKanojia no idea really, sorry! But it seems to be no easy feat: testcontainers/testcontainers-java#2857 (comment)

[tfactor2]

Did some investigation in the scope of #1502, see my comment there:
#1502 (comment).
In short, there is a way to force the plugin to use BuildKit. But it doesn't bring any value as legacy and BuildKit APIs are not compatible, see testcontainers/testcontainers-java#2857 (comment).

[rhuss]

@tfactor2 thanks for your investigations, really helpful ! I also think that having multi-arch support (for which you do need buildkit) would be a very useful feature to have (not only to support Apple's M1 but also for RasPi based builds).

In addition, BuildKit also helps in generating the proper Docker manifest so that we could push multi-arch images to registries).

I don't really know the BuildKit (or 'version 2' API) but I think it would be a worthful task to tackle, that I would split into two parts:

• Building with BuildKit
• Pushing multiple images for multiple archs to a registry with a manifest

d-m-p already has support for a second build mechanism (via JIB), so theoretically it should not be impossible to hook in a third build mechanism via BuildKit with its new API.

Unfortunately, I have zero time to investigate or implement this interesting use case :-( but I'm supportive of any questions (saying this while already being miles away from the code for quite some time). But I'm pretty sure we would get a PR integrated, together with @rohanKanojia who actually is the maintainer of this plugin. However, this only works with somebody who is fancy to step up for the implementation.

[tfactor2]

Looks like the BuildKit API turned to be very complicated to use. Python guys decided to don't implement it docker/docker-py#2230 (docker-py library), but rather wrap docker cmd with python (python-on-whales library, see https://gabrieldemarmiesse.github.io/python-on-whales/#why-another-project-why-not-build-on-docker-py).

For me JIB has a major disadvantage - it doesn't use Dockerfile but rather its own DSL to describe images. Dockerfile is a standard de-facto, no reason to learn yet another DSL IMHO.

[rhuss]

Looks like the BuildKit API turned to be very complicated to use. Python guys decided to don't implement it docker/docker-py#2230 (docker-py library), but rather wrap docker cmd with python (python-on-whales library, see https://gabrieldemarmiesse.github.io/python-on-whales/#why-another-project-why-not-build-on-docker-py).

Well, if this is the case, I wouldn't mind to do the same for multi-arch builds. We have a nice abstraction within d-m-p for doing the builds, so having an implementation that calls out to the CLI is not something out of scope. Of course, there should be still support for v1 via a remote rest API (TCP/unix-socket), but nobody prevents us from requiring a builkit installation for more advance operations (like multi-arch builds)

For me JIB has a major disadvantage - it doesn't use Dockerfile but rather its own DSL to describe images. Dockerfile is a standard de-facto, no reason to learn yet another DSL IMHO.

True, but if it fits perfectly the 'standard java app' which has three layers: base image, dependencies, your code. It's very restricted what you can do, but for many use cases starting up a java process is good enough. What you get from JIB is much faster image build and push times (because the dependency layer rarely changes). But your mileage may vary, still, you have the choice.

[tfactor2]

For layering, we are using Spring Boot layers thus JIB is not that useful for us.

[agudian]

We looked into integrating cross-platform Docker build capabilities into the docker-maven-plugin over the last two days during a small hackathon, and I want to share what we learned so far:

Using buildx instead of build

Working with buildx requires booting up a proper builder, e.g. using docker buildx create --driver docker-container --name xxx, which a developer may or may not want to re-use. For a good dev-experience, this would need to be integrated into the plugin, similar to how the docker-machine was created/terminated if required. That's a bit of work, but can be done, and as a workaround you'd just start a builder container before launching the maven build.

Building multi-arch images with buildx works great - but they are then residing inside that builder container - not where we can actually start the image, or add tags or so. Instead, buildx allows you to either --load the created image from the builder into your "host" docker environment, or it can directly --push the images to the registry.
Unfortunately, when using --load, you can only build for one platform. And many of the internals of dmp use the single imageId to refer to one created image.

In our expriment, we hooked in a command-line invocation of docker buildx build into the BuildService to build requested platforms in parallel, then running the command again without the platform parameter and with --load to load the (cached) image from the builder to the hosting docker environment - simulating what the docker:build goal usually creates, for usage in follow-up docker:start invocations within the build. A bit hacky, but it worked.
But it gets only messier after that: usually we also want to push the images to the registry. For that, we can't use the imageId that d-m-p hands around. We have to run docker buildx build again, now with the --push option to push the hopefully cached images to their registry.

Worse: we can't pass on the credentials from the nice credential chain that d-m-p uses (neither for pulling base images, nor for pushing), we can only rely on the docker login that needs to have happened outside of the Maven build.

All in all, the buildx semantics don't really fit the flow that's currently used within the d-m-p goals to build, start, tag, push. Limitations in the capabilities of buildx (for passing credentials, for exporting all created images AND also exporting a proper list-manifest) are tedious to work around.

Using docker build --platform with single platforms

We also experimented with splitting up the platform-list to run single builds with only one platform at a time. Docker automatically switches to use a buildkit based builder underneath when it encounters a non-native platform, which makes that easy. However, the platform parameter in the Docker API seems to be ignored in that case - we only got it to work with using the CLI docker build --platform ... from within d-m-p.
With docker manifest create (only available on CLI, not on the API), we could create the list-manifest on our own, e.g. when attempting to run the push-goal, and then push the whole bunch. That might be better to integrate in the current flow of the d-m-p, but requires more tags to be pushed and again, as it involves pushing and building on the command line, does not support the credential chain of the plugin.

Possible Alternatives

As trying to mix the buildx semantics with the current goals seems to be very hacky, it might be better to expose that change in semantics to the user by adding a new (set of) buildx-goal(s), with options exposed that reflect the circumstances better. For example, to manage the builder container. Or to specify which platform's image one wants to expose to the local docker environment.
Or, dreaming here, creating a local registry endpoint that buildx pushes all images and list-manifests to, which then automatically exposes those images to the local environment (e.g. by pulling from that localhost registry when the image is referenced in a "start" goal, in a "from", or whatever... ).

I wish I was able to just hand you a PR, saying "look, we integrated the calls to buildx, it works now"! But unfortunately, it's not just an issue of an API incompatibility, but an issue of a different workflow that is required.

[rhuss]

Hey, @agudian that's an awesome investigation of the situation and very very helpful ! Thanks a ton !

I agree that probably having a separate mvn docker:buildx goal would be the way to go, as it also matches the user experience that Docker itself has chosen (having two different build implementations, probably for some reason). (although having a single abstraction would have been super cool :)

wrt/ the challenge to have the built image available for e.g. mvn docker:run without pushing it to a registry first: I wonder whether we could leverage the /images/load API endpoint to load the image directly into the Docker daemon at hand (much like buildx --load would probably do).

Creating a local registry is probably out of scope, except that there would be already a Java library that would handle that. Registries are hard to implement correctly.

Said all that, unfortunately, I can't really invest time in helping to implement that (as much as I wanted it, but I'm currently totally overloaded with completely different work). But we (@rohanKanojia and I) are happy to help to integrate PRs that go into this direction.

[qerub]

I just discovered today that the /build API endpoint has a version parameter since version 1.38 where the value 2 enables BuildKit. See https://docs.docker.com/engine/api/v1.38/#tag/Image/operation/ImageBuild. This wasn't documented until February this year: moby/moby#47259