Extension Installing Fails because of duckdb build git release version

[ramarnat]

Installing Extensions using SELECT duckdb.install_extension('azure'); fails.

WARNING:  (duckdb_install_extension) HTTP Error: Failed to download extension "azure" at URL "http://extensions.duckdb.org/17d598fc44/linux_amd64/azure.duckdb_extension.gz" (HTTP 403)
Extension "azure" is an existing extension.

The reason for this is the 17d598fc44, which is the git release for duckdb. I have tried using OVERRIDE_GIT_DESCRIBE=v0.1.1 GEN=ninja make to have duckdb built with the right version, but that didnt work.

Any recommendations on how to make sure the duckdb version is overridden?

[wuputah]

try OVERRIDE_GIT_DESCRIBE=v1.1.0 😁

[ramarnat]

ha. sorry that was a copy pasta, but it is v1.1.0 🤷🏽‍♂️

I would have expected the failure to show the overriden value not the git release from the git.

[wuputah]

Hmm I see, yeah, I'm not sure how this works exactly. Something we will need to solve for functional builds / binaries. @Tishj do you know?

[wuputah]

oh may be because the .gitmodules hash is wrong, you should be on fa5c2fe15f3da5f32397b009196c0895fce60820... let me send a PR.

I really hate working with git submodules!

[ramarnat]

I dont think that's the issue, because I am doing this in Docker with a specific version of duckdb, if I was using that git hash, the error would be: at URL "http://extensions.duckdb.org/**fa5c2fe15f3d**/linux_amd64/azure.duckdb_extension.gz" (HTTP 403)

RUN --mount=type=ssh git submodule update --init --recursive --recommend-shallow && \
    cd third_party/duckdb && \
    git fetch --all && \
    git checkout $DUCKDB_VERSION

[wuputah]

Exactly, the URL is wrong. Try using my branch, or applying #185, if you can, or wait until that's merged.

[ramarnat]

I see that the azure extension is available at

http://extensions.duckdb.org/fa5c2fe15f/linux_amd64/azure.duckdb_extension.gz

but its also available at

http://extensions.duckdb.org/v1.1.0/linux_amd64/azure.duckdb_extension.gz

And I'd like to be able to checkout a named tag and have that be used in the duckdb call....

[wuputah]

Right, those two commits are the same. 👍

But otherwise, yeah I get the point. I'm not sure how to accomplish that, that's a "building duckdb" question I don't have an answer to, but it's not specific to pg_duckdb.

[wuputah]

After further investigation in #185, that hash value has no effect. You should be able to get extensions for fa5c2fe15f3d, as that is the commit for v1.1.0. If you are not getting that hash, then I think your git submodule is not up to date with main.

I'll leave this open to see if we can advise on how to get v1.1.0 in the URL instead.

[ramarnat]

Apologies, my initial attempt didnt set the build variable OVERRIDE_GIT_DESCRIBE correctly, and it does work. Anyway here is the Dockerfile that works and sets up the build using the tag. The Makefile used in the install section excludes the dependency check.

ARG PG_MAJOR=16
ARG DUCKDB_VERSION=v1.1.0
FROM postgres:$PG_MAJOR AS pg_duckdb
ARG PG_MAJOR
ARG DUCKDB_VERSION

RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
    --mount=target=/var/cache/apt,type=cache,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends \
    bison \
    build-essential \
    ca-certificates \
    cmake \
    flex \
    g++ \
    gfortran \
    git \
    libc++-dev \
    libc++abi-dev \
    libglib2.0-dev \
    liblz4-dev \
    libreadline-dev \
    libssl-dev \
    libstdc++-12-dev \
    libtinfo5 \
    libxml2-dev \
    libxml2-utils \
    libxslt-dev \
    make \
    musl-dev \
    ninja-build \
    openssh-client \
    pkg-config \
    postgresql-server-dev-$PG_MAJOR \
    xsltproc \
    zlib1g-dev && \
    mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

COPY . /pg_duckdb/
WORKDIR /pg_duckdb
RUN --mount=type=ssh git submodule update --init --recursive --recommend-shallow && \
    cd third_party/duckdb && \
    git checkout $DUCKDB_VERSION

RUN --mount=type=ssh --mount=target=/pg_duckdb/third_party/duckdb/build,type=cache \
    # rm -rf /pg_duckdb/third_party/duckdb/build/* && \
    # GEN=ninja make -j2
    OVERRIDE_GIT_DESCRIBE=$DUCKDB_VERSION GEN=ninja make 

FROM postgres:$PG_MAJOR AS postgres
ARG PG_MAJOR

RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
    --mount=target=/var/cache/apt,type=cache,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends \
    make \
    ca-certificates

COPY ./Makefile.install /tmp/Makefile
RUN --mount=type=bind,from=pg_duckdb,source=/pg_duckdb,target=/pg_duckdb,rw \
    --mount=type=bind,from=pg_duckdb,source=/usr/lib/llvm-16,target=/usr/lib/llvm-16 \
    --mount=target=/pg_duckdb/third_party/duckdb/build,type=cache \
    cd /pg_duckdb && cp /tmp/Makefile . && make install