<jemalloc>: Unsupported system page size with cockroach 23.1.5

[fveauvy]

Describe the problem

I'm getting an error when running cockroachdb v23.1.5 linux-arm64. I'm using an Asahi Linux kernel (6.3.0-asahi-10-1-ARCH) running on a Mac M1 mini.

Cockroach fails with :

<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
runtime/cgo: malloc failed: Cannot allocate memory

Note that I don't get this error using the version 22.2.11 Docker image : cockroachdb/cockroach:arm64-v22.2.11

To Reproduce

Using Asahi Linux on a Apple Silicon Mac :

1. docker pull cockroachdb/cockroach:23.1.5
2. docker volume create roach-single
3. docker run -it -v "roach-single:/cockroach/cockroach-data" cockroachdb/cockroach:23.1.5 start-single-node --insecure
4. You'll see the error mentioned above in the console output

Expected behavior

Since cockroach 22.2.11 is working fine on linux/arm64 with Linux Asahi, I think it should also run with 23.1.5.

Additional data / screenshots

Here are some related links and issues :

• storage: panic in block cache refcnt.{acquire,release} on arm64 #84971 (comment)
• https://asahilinux.org/2022/03/asahi-linux-alpha-release/
• Compiled-in page size makes jemalloc binaries unportable jemalloc/jemalloc#467
• Use jemalloc by default also on ARM redis/redis#11407
• Binary releases of fd crash on Asahi Linux because of jemalloc using 8kb page sizes sharkdp/fd#1085

Additional context

We're migrating our CI/CD pipeline to use GitHub Actions Self-Hosted Runners. For some reasons, we want to use Linux and our available Mac Mini M1 in order to run testing workflows that includes CockroachDB services.

Jira issue: CRDB-29697

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

• @jbowens (commented on storage: panic in block cache refcnt.{acquire,release} on arm64 #84971)
• @sumeerbhola (commented on storage: panic in block cache refcnt.{acquire,release} on arm64 #84971)
• @itsbilal (commented on storage: panic in block cache refcnt.{acquire,release} on arm64 #84971)
• @RaduBerinde (assigned to storage: panic in block cache refcnt.{acquire,release} on arm64 #84971)
• @nicktrav (commented on storage: panic in block cache refcnt.{acquire,release} on arm64 #84971)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[RaduBerinde]

To fix this, we need to configure jemalloc to support a larger page size (--with-lg-page=16). However, in some cases that setting caused problems, at least a while back - #81581.

[RaduBerinde]

@fveauvy I will try to tweak the jemalloc build settings for ARM to support larger pages. I do want to point out though that for now ARM support is best-effort and we don't recommend using it for production. We have filed internal tickets to improve the documentation to be more clear about this.

[fveauvy]

@fveauvy I will try to tweak the jemalloc build settings for ARM to support larger pages. I do want to point out though that for now ARM support is best-effort and we don't recommend using it for production. We have filed internal tickets to improve the documentation to be more clear about this.

@RaduBerinde Duly noted. As I mentioned in additional context, we don't plan on using it for production purposes but for our unit/integration test environment with GitHub Actions self hosted runners.

Thanks a lot for your input.

[RaduBerinde]

@fveauvy Hopefully the next 23.1.x release works.

[pawalt]

FYI this is fixed for me on the most recent version of CRDB. I'm also on Asahi kernel 6.5.0-asahi. This is the command that passed for me:

sudo docker run -it -v "roach-single:/cockroach/cockroach-data" cockroachdb/cockroach:v23.1.13 start-single-node --insecure

Thanks for the help @RaduBerinde

[RaduBerinde]

Thanks @pawalt! I will close this then.