caching with COPY --from

[virth]

Actual behavior

Kaniko uses a the cached version of a COPY --from command although the copied source has changes in it and shouldn't be cached.

Disclaimer: I'm not entirely sure if this is a bug or if the "correct" caching of a COPY --from ... command is simply a feature that's not supported yet.

In a first step we build a "base" docker image with all our dependencies that gets pushed to GCR so we can use this image for multiple steps later. When we pull the previously built image as the base for our next step, we copy the content of the base image into another stage uf the multistage docker build. Kaniko detects that there is a cached Layer for this and uses this instead of the actual new, changed source.

Expected behavior
A clear and concise description of what you expected to happen.

To Reproduce
Steps to reproduce the behavior:

1. Enable --cache for all your builds.
2. Build a docker image that you push to GCR with GCB and Kaniko.
3. Pull the previous built image (as a base image) in your second job and use COPY --from base ./app . to copy the content of the first image.
4. Run the entire pipeline again but change something in your first image.
5. Kaniko will use the cached Layer of the COPY --from base ./app . altough there were changes

Additional Information

• 1st Dockerfile

FROM node:10-alpine as dependencies

WORKDIR /app
COPY ./package.json ./package-lock.json .
COPY ./src .

RUN npm run ci

FROM alpine:3.9 as final

COPY --from=dependencies /app /app

• 2nd Dockerfile

FROM my-base-image:latest as base
FROM testcafe/testcafe:1.8.4 as test

WORKDIR /app
COPY --from=base /app .

RUN npm run test:integration:ci 

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

Please let me know if you need anything else !

[carno]

I can see a similar issue with kaniko 0.24.0 and 0.23.0.

0.18.0 seems to work as expected (but maybe it's just the case of it missing the cache more in general)

[tejal29]

@virth, theoretically, in case of the 2nd dockerfile,

FROM my-base-image:latest as base.  # <-- THE HASH calculated for this image should if 
                                    # 1st dockerfile has a change and pushed a new image
FROM testcafe/testcafe:1.8.4 as test

WORKDIR /app
COPY --from=base /app .  # if above happens, then the hash computed for caching for this 
                         # line should also change resulting in a cache miss and updated /app should be copied.

RUN npm run test:integration:ci 

Can you provide us with trace logs?

[gsaraf]

We are encountering this problem now. Here is a sample Dockerfile:

FROM alpine:3.13 as builder
COPY some_changing_file /some_changing_file

FROM alpine:3.13
COPY --from=builder /some_changing_file /some_changing_file
CMD cat /some_changing_file

Changing the value of /some_changing_file does not change the output of the cat in the final built container. It "remembers" the value from the very first time it was built, no matter what changes are made to the builder.

I've attached trace logs indicating the problem below. See lines 47 for the some_changing_file to not be cached, and line 806 for using the cached version of the COPY --from.
kaniko_bug_trace.log

I think this is clearly a bug, since docker build behaves as expected, and does not cache when it shouldn't.

I'd be happy to attempt to solve this.

[KasnocknDave]

We are experiencing the same problem as described above when building a Go project, which led us to deactivating the --cache flag globally. This results in performance issues when combined with some of our more complex builds, since they need to install lots of rarely changing dependencies in builder stage - especially with node_modules.

Our example:

FROM golang:1.15 as builder
WORKDIR /app
COPY ./go.sum ./go.mod ./
RUN go mod download
COPY ./resources ./resources
RUN CGO_ENABLED=0 go build -o go-service main.go

FROM alpine:3.12.3
WORKDIR /app
COPY  --from=builder /app/go-service /app
CMD ["./go-service", "p", "-app-path=/app"]

Additionally we ran a script to create a version number within the build for every new image. The ouput of this script showed that a new version was indeed created and saved into a file, but when loading it in the second stage suddenly the first ever cached version was used, matching the experience of the devs posting above me.

This was tested with the latest executor version (1.6.0).

[pblgomez]

Still happening? Is there going to be a fix for this?

[aanthonyrizzo]

Hi all! I recently ran into this issue. It seems this was fixed in #1735 and then regressed. I downgraded my kaniko from 1.8 => 1.7 and it seemed to work for me.

[aaron-prindle]

I believe this has been fixed with #2559.

I attempted the repro here:
#1348 (comment)

with caching enabled and was seeing my changes represented in each created image, ex:

aprindle@aprindle-ssd ~/kaniko-ctx-dir cat Dockerfile; echo
FROM alpine:3.13 as builder
COPY orig-file.txt some-changing-file.txt

FROM alpine:3.13
COPY --from=builder some-changing-file.txt some-changing-file.txt
CMD cat some-changing-file.txt
aprindle@aprindle-ssd ~/kaniko-ctx-dir cat orig-file.txt; echo # note this file was changed from one -> two -> three over three cached runs
three

$ ./run_in_docker.sh /workspace/Dockerfile /usr/local/google/home/aprindle/kaniko-ctx-dir gcr.io/aprindle-test-cluster/kaniko-test:latest true
...
DEBU[0001] Build: cache key for command CMD cat some-changing-file.txt e3247dcec59319f30529daacd1e6d230b04c63c0d4ff7536289c0db3865273b6 
DEBU[0001] Mapping stage idx 1 to digest sha256:0dd9f0f14054dad604b1f1c6b2332d501a0917ffa4de9c8a73f50540a868474a 
DEBU[0001] Mapping digest sha256:0dd9f0f14054dad604b1f1c6b2332d501a0917ffa4de9c8a73f50540a868474a to cachekey e3247dcec59319f30529daacd1e6d230b04c63c0d4ff7536289c0db3865273b6 
INFO[0001] Pushing image to gcr.io/aprindle-test-cluster/kaniko-test:latest 
INFO[0004] Pushed gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec 
aprindle@aprindle-ssd ~/kaniko  [main]docker pull  gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec 
^C
aprindle@aprindle-ssd ~/kaniko  [main]docker pull  gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec 
gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec: Pulling from aprindle-test-cluster/kaniko-test
72cfd02ff4d0: Pull complete 
64c27253fcaa: Pull complete 
Digest: sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec
Status: Downloaded newer image for gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec
gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec
aprindle@aprindle-ssd ~/kaniko  [main]docker run -it  --entrypoint=/bin/sh gcr.io/aprindle-test-cluster/kaniko-test@sha256:2802586c6d3ec06a504f06f69f3220663c62a493b804dfb0fde3988d5db5b2ec 
/ # ls
bin                     media                   run                     tmp
dev                     mnt                     sbin                    usr
etc                     opt                     some-changing-file.txt  var
home                    proc                    srv
lib                     root                    sys
/ # cat some-changing-file.txt 
three # <--- this is the most up to date change, not seeing cached version of file
/ # 

[aaron-prindle]

If anyone in the thread here can confirm this has been fixed w/ Kaniko v1.12.1 it would be much appreciated. If you are still seeing this issue, comment and I can re-open. Thanks

[kalda341]

I'm experiencing what I think is the same issue with v1.15.0-debug.

I'm building two images within, one after the other with the following command (repeated twice with different PROJ, TARGET and TAG):

/kaniko/executor
  --context "${CI_PROJECT_DIR}"
  --dockerfile "${CI_PROJECT_DIR}/packages/${PROJ}/Dockerfile"
  --target "$TARGET"
  --destination "${TAG}"

Both Dockerfiles are very similar, and have a prune stage, defined as:

FROM base AS prune
WORKDIR /app
COPY . .
RUN pnpm dlx turbo prune --scope=@rico-technologies/${PROJ} --docker

The second build is using the first prune stage, and not the one defined in the second Dockerfile.
Given the way these two projects work, I believe the second prune stage contains a subset of the files produced by the first - could this be why the caching doesn't work?

Also, note that this is WITHOUT any cache flags enabled!