Docker image

[wuputah]

Starting point for a Postgres image with pg_duckdb installed.

[owenthereal]

For context reference, the reason that we can't build hydra using pgxman as part of its CI is due to the intricate limitation of pgxman's publication mechanism. It necessitates the existence and publication of the buildkit file in a central repo pgxman/buildkit, which in turn refers to a tag for source download. This creates a circular dependency, where the tagging of an extension requires pgxman to build the buildkit artifacts, but the build artifacts require the tag to exist first. This can be worked around now by:

1. Create a buildkit in this repo with source pointing to a local path (this is already working but undocumented). This would generate the Debian files to copy into a Docker image. Even better, add a command pgxman build --export DOCKER_IMAGE that does both.
2. For publication, we duplicate the same buildkit and save it to pgxman/buildkit with source pointing to the source of a tag. Having two canonical copies of buildkit files is not ideal, but it's not the end of the world.

When we can self-publish extensions from individual repo in the future, the circular dependency will be broken, and this workaround will no longer be necessary.

[wuputah]

heya @owenthereal, sorry for the lack of context, I added you here in part because of pgxman but also because I'm a bit of a noob when it comes to Docker so I wanted you to check my work, if you had any suggestions for improvement. For instance I'm current creating a checker image to run the tests but I guess Docker determines that this isn't necessary to make the final image, so it skips this step. Of course I could just run the tests in builder instead.

I do think it would be possible to build with pgxman based on the docs here, as you suggest a buildkit yaml would be needed as well.
https://docs.pgxman.com/building_an_extension#test-the-extension

IMO this would just be for making "dev builds" as desired, though for local testing the duckdb build takes a long time and the ccache setup seems to not work super well, so it's not a great local developer tool.

[owenthereal]

For instance I'm current creating a checker image to run the tests but I guess Docker determines that this isn't necessary to make the final image, so it skips this step.

You could run a specific target with https://docs.docker.com/reference/cli/docker/image/build/#target, e.g. docker build ... -t checker

though for local testing the duckdb build takes a long time and the ccache setup seems to not work super well

Not saying we should replace this Docker build with pgxman build right now, but being able to specify cache dir would be a nice pgxman feature to add in the future.

[mike-luabase]

this is what ultimately worked for me:

FROM postgres:16-bookworm as base

###
### BUILDER
###
FROM base as builder

RUN --mount=type=cache,target=/var/cache/apt \
  apt-get update -qq && \
  apt-get install -y build-essential libreadline-dev zlib1g-dev flex bison libxml2-dev libxslt-dev \
  libssl-dev libxml2-utils xsltproc pkg-config libc++-dev libc++abi-dev libglib2.0-dev libtinfo5 cmake \
  libstdc++-12-dev postgresql-server-dev-16 liblz4-dev ccache git

WORKDIR /build

ENV PATH=/usr/lib/ccache:$PATH
ENV CCACHE_DIR=/ccache

# Clone the pg_duckdb repository and initialize submodules
RUN git clone --branch main https://github.com/duckdb/pg_duckdb.git . && \
    git submodule update --init --recursive

# permissions so we can run as `postgres` (uid=999,gid=999)
RUN chown -R postgres:postgres .
RUN chown -R postgres:postgres /usr/lib/postgresql /usr/share/postgresql
RUN mkdir /out && chown postgres:postgres /out
RUN rm -f .depend

USER postgres

# Build and install
RUN --mount=type=cache,target=/ccache/,uid=999,gid=999 make install
RUN --mount=type=cache,target=/ccache/,uid=999,gid=999 DESTDIR=/out make install

###
### CHECKER
###
FROM builder as checker

USER postgres
RUN --mount=type=cache,target=/ccache/,uid=999,gid=999 make installcheck

###
### OUTPUT
###
# This creates a usable postgres image but without the packages needed to build
FROM base as output
COPY --from=builder /out /

[jorinvo]

I would love to install pg_duckdb in our Postgres Docker image. I tried the Dockerfile from above, but it was stuck at 100% at make install for 30 minutes.

[mike-luabase]

I would love to install pg_duckdb in our Postgres Docker image. I tried the Dockerfile from above, but it was stuck at 100% at make install for 30 minutes.

Took a very long time to run for me too. Might have been more than 30 minutes before it was complete.

[jorinvo]

Thanks @mike-luabase, that's good to know.
I am afraid that's not usable for us for now. But I am looking forward to having a prebuilt image available some day. pg_duckdb is definitely an exciting project 🤩

[JelteF]

To make the build faster it would help a lot if you changed the make install commands to be parallel, by using e.g. make -j10 install. Or maybe make -j$(nproc) install

[wuputah]

To make the build faster it would help a lot if you changed the make install commands to be parallel, by using e.g. make -j10 install. Or maybe make -j$(nproc) install

Even after #211, there remains an issue with -j being passed to the duckdb build that I haven't managed to solve (though I tried). You'll see this printed in the log when running make duckdb:

make[2]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

This is despite the fact we are using $(MAKE) as suggested for this issue. This probably has something to do with DuckDB's use of cmake, but I don't know anything about cmake.