Pillow installed via Conda on Linux errors with libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0

[kevinrue]

What did you do?

I am debugging an issue for a Bioconductor R package that doesn't build on Linux at the moment:
https://bioconductor.org/checkResults/devel/bioc-LATEST/velociraptor/nebbiolo2-buildsrc.html

I have replicated the issue on a HPC at work. However, the HPC does not provide the version of R that I need, I have followed IT guidelines and used Apptainer to work in a container that provides everything I need to replicate the issue.

The issue arose when I ran R CMD build on the Bioconductor R package that uses a Conda environment to run some Python code internally.

apptainer shell \
  --writable-tmpfs \
  -B /project/sims-lab/albrecht/git/kevinrue/velociraptor \
  -B /project/sims-lab/albrecht/gypsum \
  -B /ceph/project/sims-lab/albrecht/R-cache \
  /project/sims-lab/albrecht/mycontainers/velociraptor.sif

# -B /project/sims-lab/albrecht/gypsum \ needed to download and access test data
# -B /ceph/project/sims-lab/albrecht/R-cache \ needed to set up the conda environment using basilisk

# needed to access data during R CMD build
export GYPSUM_CACHE_DIR="/project/sims-lab/albrecht/gypsum"

# R CMD build
R CMD build velociraptor

What did you expect to happen?

The package used to pass R CMD build, e.g. https://bioconductor.org/checkResults/3.18/bioc-LATEST/velociraptor/nebbiolo2-install.html

What actually happened?

The R function that creates the Conda environment and runs the Python code crashes with the error

ImportError: /ceph/project/sims-lab/albrecht/R-cache/R/basilisk/1.17.0/velociraptor/1.15.2/env/lib/python3.9/site-packages/PIL/../../../libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0

Which I have seen in the previous issue #7269 but cannot find a way to fix in my case.

What are your OS, Python and Pillow versions?

• OS: Linux
• Python: 3.9.12
• Pillow: 10.3.0

Please paste here the output of running:

python3 -m PIL.report
or
python3 -m PIL --report

Or the output of the following Python code:

from PIL import report
# or
from PIL import features
features.pilinfo(supported_formats=False)

--------------------------------------------------------------------                                                                                        
Pillow 10.3.0                                                                                                                                               
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:34:28)                                                                                     
       [GCC 10.3.0]                                                                                                                                         
--------------------------------------------------------------------
Python executable is /ceph/project/sims-lab/albrecht/R-cache/R/basilisk/1.17.0/velociraptor/1.15.2/env/bin/python
System Python files loaded from /ceph/project/sims-lab/albrecht/R-cache/R/basilisk/1.17.0/velociraptor/1.15.2/env
--------------------------------------------------------------------
Python Pillow modules loaded from /ceph/project/sims-lab/albrecht/R-cache/R/basilisk/1.17.0/velociraptor/1.15.2/env/lib/python3.9/site-packages/PIL
Binary Pillow modules loaded from /ceph/project/sims-lab/albrecht/R-cache/R/basilisk/1.17.0/velociraptor/1.15.2/env/lib/python3.9/site-packages/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.3.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.12.1 
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.4.0
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.0
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.11
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
--- XCB (X protocol) support ok
--------------------------------------------------------------------

[radarhere]

In #7269, the undefined symbol was in _imaging.so, so I do think that's different.

Allow me to try and understand - you've pasted the features/report info, showing the Pillow installation and the versions of its dependencies. So you are able to install Pillow successfully. What you're talking about is something else importing Pillow, and that triggers the error.

that doesn't build on Linux at the moment

So it did build on Linux previously at https://bioconductor.org/checkResults/3.18/bioc-LATEST/velociraptor/nebbiolo2-install.html, but now fails at https://bioconductor.org/checkResults/devel/bioc-LATEST/velociraptor/nebbiolo2-buildsrc.html.

Any idea what triggered this change?

to run some Python code internally.

From a quick browse of https://github.com/kevinrue/velociraptor, is scvelo where this becomes a Python matter? So was this triggered by the recent kevinrue/velociraptor#71?

Ultimately, I'm not sure that we'll be able to fix this from our end. It sounds like the error is in how libtiff was built, and we don't control the build process of conda-forge pillow - that is maintained over at https://github.com/conda-forge/pillow-feedstock. Or maybe it is even a task for https://github.com/conda-forge/libtiff-feedstock, since they will willing to deal with conda-forge/libtiff-feedstock#78

[kevinrue]

Allow me to try and understand

I think you've summarised it beautifully.

Any idea what triggered this change? / was this triggered by the recent kevinrue/velociraptor#71?

Yes. The main thing that changed is the Conda environment, i.e. package versions. At least the OS is the same, and the Bioconductor core team who maintains the build system hasn't flagged any obvious change in the system. They seem as puzzled as I am.

You will see in the last commit before the PR that - when things worked - we used to pin a lot less dependencies (https://github.com/kevinrue/velociraptor/blob/dbc6491d830ec195ebe7ea14412157c7e3fb3f74/R/basilisk.R#L2-L72).
The motivation for reviewing the dependencies was that:

• The old environment specs were not working any more on MacOS Arm
• Our primary dependency scvelo was now quite a few versions ahead, and updating it required to update many of its own dependencies.
• Pinning everything explicitly would improve reproducibility (I am trying to reproduce the last successful build using an image of Bioconductor 3.18 and I'm still waiting for Conda to resolve the old environment).

is scvelo where this becomes a Python matter?

Yes

It sounds like the error is in how libtiff was built / Or maybe it is even a task for https://github.com/conda-forge/libtiff-feedstock

Sounds like I better open an issue there, linking to this one, and see what they say. Thanks a lot for your time and thorough review!

[radarhere]

If you just run

from PIL import Image
im = Image.new("RGB", (1, 1))
im.save("out.tif", compression="jpeg")

does the error occur?

[kevinrue]

Our Linux cluster is offline for maintenance today and tomorrow. I'll test this on Friday if everything is back online.
Thanks!

[kevinrue]

Actually, while waiting for the cluster to come back online, I've used GitHub Action to run the code you gave above. The error does not occur:

https://github.com/kevinrue/velociraptor/actions/runs/9594803654/job/26458166745#step:18:29

Just for the record, proof that I've run that command in the environment that raises the error undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0:

https://github.com/kevinrue/velociraptor/actions/runs/9594803666/job/26458166748#step:17:1966

[kevinrue]

Did the information above help?

[radarhere]

Having a GitHub Actions reproduction is definitely helpful. I haven't had as much time as I would like to investigate this, but here are my thoughts so far.

The fact that you're able to install and run Pillow code that uses libtiff without an error points to this not actually being a Pillow problem. I understand that the error message mentions Pillow, but I'm not sure this will be able to be replicated without scvelo.

Even if you can't replicate it without scvelo, it would be helpful if you could trigger the problem starting from Python, rather than from velociraptor.

[kevinrue]

Even if you can't replicate it without scvelo, it would be conda-forge/libtiff-feedstock#104 (comment) if you could trigger the problem starting from Python, rather than from velociraptor.

Good point. I've reproducing the error the only I know how, but I'll see if I can figure out a way to trigger the problem directly from Python. (I'm also short on time)

Many thanks for your time!

[kevinrue]

This micromamba environment and Python script simulate the workflow but do not trigger the issue from Python while it still occurs from velociraptor:
kevinrue/velociraptor#78 (comment)

[kevinrue]

In the end I think there's nothing wrong with pillow.

The issue seems to relate to a difference in environment between the R CMD build and R CMD check commands.

This command fails

remotes::install_local("/project/sims-lab/albrecht/git/kevinrue/velociraptor", dependencies = TRUE, repos = BiocManager::repositories(), build_vignettes = TRUE, upgrade = TRUE, force = TRUE, lib = "/project/sims-lab/albrecht/scvelo-test/test-velociraptor/bioc_320")

While this one works, and subsequently runs the velociraptor code fine in an interactive session.

remotes::install_local("/project/sims-lab/albrecht/git/kevinrue/velociraptor", dependencies = TRUE, repos = BiocManager::repositories(), build_vignettes = FALSE, upgrade = TRUE, force = TRUE, lib = "/project/sims-lab/albrecht/scvelo-test/test-velociraptor/bioc_320")

The only difference being build_vignettes = TRUE|FALSE meaning that for some reason the libtiff symbol is not found during R CMD build.

This SO thread points to the plausible reason, but isn't enough for me to fix my situation.

Thanks for your help. I think you can close this issue now.

[radarhere]

Thanks very much for your further investigation.

[kevinrue]

For what it's worth, I've finally sidestepped the issue by adding anaconda (first) in the channels.

Interestingly, it's reducing the number of conda packages in the environment by 57.

Not trying to reopen this issue, but in case you're interested you can compare the environment that causes me problems (without anaconda)
https://github.com/kevinrue/velociraptor/blob/devel/inst/envs/Linux_x86_64.yaml
vs the one that finally got me going again (with anaconda)
https://github.com/kevinrue/velociraptor/blob/devel/inst/envs/Linux_x86_64_anaconda.yaml

The version of pillow is the same in both cases. No clue what's causing/fixing the issue.