Caching fails in multi-stage build

[apenney]

Actual behavior
When I build a multi-stage docker file the cache is blown every time, no matter what I do. The dockerfile (first two stages, where I see the problem, there are more stages):

FROM python:2.7-slim AS base

# Make man page directories so that postgresql-client can install
RUN seq 1 8 | xargs -I{} mkdir -p /usr/share/man/man{} \
    && apt-get update \
    && apt-get install -y curl \
    && curl -sL https://deb.nodesource.com/setup_10.x | bash - \
    && apt-get install -y build-essential \
                          checkinstall \
                          default-libmysqlclient-dev \
                          libbz2-dev \
                          libffi-dev \
                          libgflags-dev \
                          libgflags2v5 \
                          liblz4-dev \
                          libncurses5-dev \
                          libpq-dev \
                          libsnappy-dev \
                          libsnappy1v5 \
                          libxml2 \
                          libxml2-dev \
                          libxmlsec1-dev \
                          libxslt1.1 \
                          libzstd-dev \
                          libzstd1 \
                          mariadb-client \
                          mysql-client \
                          nodejs \
                          postgresql-client-9.6 \
                          unixodbc \
                          unixodbc-dev \
                          wget \
                          zlib1g-dev \
    && rm -rf /var/lib/apt/lists/*

##
## RocksDB base image
##
## We don't expect this to change frequently, so we make it a stage all by
## itself so that we never rebuild it unless we change the version of rocksdb.
##
FROM base AS rocksdb-build

RUN curl -L https://github.com/facebook/rocksdb/archive/v5.18.3.tar.gz \
  | tar -xzC /tmp \
  && PORTABLE=1 make -C /tmp/rocksdb-5.18.3 -j 4 install-shared INSTALL_PATH=/install

Each build will use the cache for the first RUN, but always result in looking for a different cache for the second RUN, no matter what I do.

I run with:

/kaniko/executor --build-arg tests=false --dockerfile=`pwd`/Dockerfile -c `pwd` --cache=true --destination=x.dkr.ecr.us-east-1.amazonaws.com/web:latest --cache-repo=x.dkr.ecr.us-east-1.amazonaws.com/build_cache

Here's a snippet from a run (sorry, messy logs as I cut and paste from jenkins):

[36mINFO�[0m[0000] Resolved base name python:2.7-slim to python:2.7-slim 
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name python:2.7-slim to python:2.7-slim 
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Resolved base name base to base              
�[36mINFO�[0m[0000] Downloading base image python:2.7-slim       
2019/05/30 16:32:43 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:86e71c1b8ab705a6ec95865a2acab81bff529a840dea7269ed485e5728fd89ec: no such file or directory 
�[36mINFO�[0m[0000] Downloading base image python:2.7-slim       
2019/05/30 16:32:43 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0000] Built cross stage deps: map[1:[/install /install] 2:[/root/.local] 3:[/app/simon/static /app/frontend /app/node_modules /app/.webpack/default.json]] 
�[36mINFO�[0m[0000] Downloading base image python:2.7-slim       
2019/05/30 16:32:43 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:86e71c1b8ab705a6ec95865a2acab81bff529a840dea7269ed485e5728fd89ec: no such file or directory 
�[36mINFO�[0m[0000] Downloading base image python:2.7-slim       
2019/05/30 16:32:43 No matching credentials were found, falling back on anonymous
�[36mINFO�[0m[0000] Checking for cached layer x.dkr.ecr.us-east-1.amazonaws.com/build_cache:56c8485b18a1a4119881dc80bbf36fa1f9315975565c43d4e68c9c4b97807d06... 
[Pipeline] sh
�[36mINFO�[0m[0001] Using caching version of cmd: RUN seq 1 8 | xargs -I{} mkdir -p /usr/share/man/man{}     && apt-get update     && apt-get install -y curl     && curl -sL https://deb.nodesource.com/setup_10.x | bash -     && apt-get install -y build-essential                           checkinstall                           default-libmysqlclient-dev                           libbz2-dev                           libffi-dev                           libgflags-dev                           libgflags2v5                           liblz4-dev                           libncurses5-dev                           libpq-dev                           libsnappy-dev                           libsnappy1v5                           libxml2                           libxml2-dev                           libxmlsec1-dev                           libxslt1.1                           libzstd-dev                           libzstd1                           mariadb-client                           mysql-client                           nodejs                           postgresql-client-9.6                           unixodbc                           unixodbc-dev                           wget                           zlib1g-dev     && rm -rf /var/lib/apt/lists/* 
�[36mINFO�[0m[0001] Skipping unpacking as no commands require it. 
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0074] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
�[36mINFO�[0m[0075] Checking for cached layer x.dkr.ecr.us-east-1.amazonaws.com/build_cache:dc634b1712d0d0e3737ab1934815008fb2076172725743f710ed08b1712706fb... 
�[36mINFO�[0m[0075] No cached layer found for cmd RUN curl -L https://github.com/facebook/rocksdb/archive/v5.18.3.tar.gz   | tar -xzC /tmp   && PORTABLE=1 make -C /tmp/rocksdb-5.18.3 -j 4 install-shared INSTALL_PATH=/install 
�[36mINFO�[0m[0075] Unpacking rootfs as cmd RUN curl -L https://github.com/facebook/rocksdb/archive/v5.18.3.tar.gz   | tar -xzC /tmp   && PORTABLE=1 make -C /tmp/rocksdb-5.18.3 -j 4 install-shared INSTALL_PATH=/install requires it. 

I can immediately run the same job after this one pushes the final container and I see:

First RUN found in cache:

[36mINFO�[0m[0001] Using caching version of cmd: RUN seq 1 8 |

Second RUN fails as always:

�[36mINFO�[0m[0074] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
�[36mINFO�[0m[0075] Checking for cached layer x.dkr.ecr.us-east-1.amazonaws.com/build_cache:bbebc1c2ca0e9048c65f6262ce5649ef10ba391927a9b84170a0ca2f0f574729... 
�[36mINFO�[0m[0075] No cached layer found for cmd RUN curl -L https://github.com/facebook/rocksdb/archive/v5.18.3.tar.gz   | tar -xzC /tmp   && PORTABLE=1 make -C /tmp/rocksdb-5.18.3 -j 4 install-shared INSTALL_PATH=/install 

I don't know why it looks for dc634b1712d0d0e3737ab1934815008fb2076172725743f710ed08b1712706fb in the first attempt and bbebc1c2ca0e9048c65f6262ce5649ef10ba391927a9b84170a0ca2f0f574729 in the second. There are no other commands between the two RUNs, except another FROM.

Is this because I don't cache the base images?

Expected behavior
I expect it to find a cached layer (the layer it pushes on the previous run) and use it for this rocksdb step.

To Reproduce
Steps to reproduce the behavior:
1.. Hopefully by taking that snippet of dockerfile and running it. I'm running kaniko via Jenkins (so using the latest kaniko:debug container) the command up top.

[potiuk]

I have the very same problem :(

[subesokun]

Same problem here. A simple multi stage setup breaks already the cache. E.g.

FROM node:12-alpine AS step1
RUN echo "Do something"

FROM step1 as step2
RUN echo "Do something else"

Kaniko always rebuilds the image for step2 in this case as it generates every time a new hash.

[jrauschenbusch]

Same here. Different calculated base image id's during subsequent executions, if e.g. a WORKDIR command is used in the step1 stage. Seems to be related to the tarball caching mechanism vs a real layer caching. But it's really hard to debug and to find the relevant code locations of the issue.

[cvgw]

There is still an edge case where this might fail related to #899