[B] Building large Docker images in Kaniko leads to OOMKilled job

[YevheniiSemendiak]

Summary

If you build a relatively large docker image, Kaniko will fail to build it since its pod will be killed by K8s due to high memory usage (even in quite large presets, say, having 10Gigs of RAM).

Steps to reproduce

1. Create a Dockerfile with the following content:

FROM neuromation/neuro-extras:21.3.19
RUN wget https://raw.githubusercontent.com/neuro-inc/platform-client-python/master/build-tools/garbage-files-generator.py && \
    python3 garbage-files-generator.py 1 7Gb

2. Launch build via neuro-extras image build -s cpu-large . image:test-build-failure
3. Observe Job was OOMKilled

Expected result

The build finishes properly.

Environment

Mandatory:

• neuro-extras version: 21.3.19
• neuro CLI version: 21.6.17
• KANIKO_IMAGE_REF = "gcr.io/kaniko-project/executor"
• KANIKO_IMAGE_TAG = "v1.5.1"

Additional information (optional)

Example job ID: job-98cf4efa-4128-49d6-9f8f-01937011ed67
Manual rerun with disabled Kaniko caching (job-c26e3874-fcfa-478f-a862-72a28394853c) leads to the same error.

[YevheniiSemendiak]

Some relevant reports in Kaniko repo:

• Using --reproducible loads entire image into memory GoogleContainerTools/kaniko#862
• Snapshot includes whole image GoogleContainerTools/kaniko#1569

[YevheniiSemendiak]

v1.6.0 with --cache=false - OOMKilled (metrics)
v1.6.0 with --cache=false and default --snapshotMode - OOMKilled (job-699d81aa-0e61-4eef-8287-e7a611906765)
v1.5.0 - OOMKilled
v1.3.0 - OK, (yet, no --cache-copy-layers flag usage), metrics

[YevheniiSemendiak]

created an issue in Kaniko repo GoogleContainerTools/kaniko#1680

[YevheniiSemendiak]

Downgraded Kaniko to v1.3.0 in #287 , need to bump when the will be fixed in Kaniko repo.

[YevheniiSemendiak]

Kaniko release 1.7.0 resolves the problem. We need to expose that flag for users and bump Kaniko.