Cannot install certain packages after h5py without breaking h5py

[mhearne-usgs]

An example script is below. Running it gives me this error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/mhearne/anaconda/envs/shake/lib/python3.5/site-packages/h5py/__init__.py", line 24, in <module>
    from . import _errors
ImportError: dlopen(/Users/mhearne/anaconda/envs/shake/lib/python3.5/site-packages/h5py/_errors.cpython-35m-darwin.so, 2): Library not loaded: @rpath/libhdf5.10.dylib
  Referenced from: /Users/mhearne/anaconda/envs/shake/lib/python3.5/site-packages/h5py/_errors.cpython-35m-darwin.so
  Reason: Incompatible library version: _errors.cpython-35m-darwin.so requires version 12.0.0 or later, but libhdf5.10.dylib provides version 11.0.0

#!/bin/sh

source activate root
conda remove --name shake --all -y
conda create --name shake --channel conda-forge python=3 -y
source activate shake
conda install -y numpy
conda install -y scipy
conda install -y matplotlib
conda install -y jupyter
conda install -y h5py

python -c "import h5py"

conda install -y rasterio

python -c "import h5py"

[ocefpaf]

@mhearne-usgs can you send the conda list results? Without looking at what the conda solver decided to install it is hard to debug.

Your error message is probably because rasterio installation changed the hdf5 package that was originally OK for h5py.

BTW that is an issue that will be solved soon in conda. The user will have options to choose channel preference and, in theory, reduce package mismatch.

PS0: Take a look at the edits I made to your original message. GitHub uses Markdown to render them and your separators are valid Markdown syntax that made it hard to read.

PS1: I suspect we'll have to re-build libnetcdf with our hdf5 to be able to solve this.

[ocefpaf]

BTW this worked for me:

conda create --name TEST --yes --channel conda-forge python=3 numpy scipy matplotlib jupyter h5py rasterio
source activate TEST
conda uninstall libnetcdf
conda install hdf5==1.8.16
python -c "import h5py"

[jakirkham]

Yeah, I'm not sure where these are all coming from. Could you please run this conda config --set show_channel_urls True and then re-run your script with a fresh environment?

[mhearne-usgs]

Here is the result from conda list:

# packages in environment at /Users/mhearne/anaconda/envs/shake:
#
affine                    1.2.0                    py35_0    defaults
appnope                   0.1.0                    py35_0    defaults
click                     6.3                      py35_0    defaults
click-plugins             1.0.3                    py35_0    ioos
cligj                     0.4.0                    py35_0    defaults
curl                      7.45.0                        0    defaults
cycler                    0.10.0                   py35_0    defaults
decorator                 4.0.9                    py35_0    defaults
expat                     2.1.0                         0    conda-forge
freetype                  2.5.5                         0    defaults
freexl                    1.0.2                         0    conda-forge
gdal                      1.11.4              np110py35_2    conda-forge
geos                      3.4.2                         1    conda-forge
giflib                    5.1.2                         0    conda-forge
h5py                      2.6.0               np110py35_1    conda-forge
hdf4                      4.2.11                        0    defaults
hdf5                      1.8.15.1                      2    defaults
ipykernel                 4.3.1                    py35_0    defaults
ipython                   4.1.2                    py35_1    defaults
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py35_0    defaults
ipywidgets                4.1.1                    py35_0    defaults
jasper                    1.900.1                       0    conda-forge
jbig                      2.1                           0    defaults
jinja2                    2.8                      py35_0    defaults
jpeg                      8d                            1    defaults
json-c                    0.12                          0    conda-forge
jsonschema                2.4.0                    py35_0    defaults
jupyter                   1.0.0                    py35_2    defaults
jupyter-client            4.2.2                     <pip>
jupyter-console           4.1.1                     <pip>
jupyter-core              4.1.0                     <pip>
jupyter_client            4.2.2                    py35_0    defaults
jupyter_console           4.1.1                    py35_0    defaults
jupyter_core              4.1.0                    py35_0    defaults
kealib                    1.4.5                         0    defaults
krb5                      1.13.2                        0    defaults
libgcc                    4.8.5                         1    defaults
libnetcdf                 4.3.3.1                       3    defaults
libpng                    1.6.17                        0    defaults
libtiff                   4.0.6                         1    defaults
libtool                   2.4.6                         0    conda-forge
markupsafe                0.23                     py35_0    defaults
matplotlib                1.5.1               np110py35_0    defaults
mistune                   0.7.2                    py35_1    defaults
mkl                       11.3.1                        0    defaults
nbconvert                 4.1.0                    py35_0    defaults
nbformat                  4.0.1                    py35_0    defaults
notebook                  4.1.0                    py35_2    defaults
numpy                     1.10.4                   py35_0    defaults
openjpeg                  2.1.0                         1    conda-forge
openssl                   1.0.2g                        0    defaults
path.py                   8.1.2                    py35_1    defaults
pexpect                   4.0.1                    py35_0    defaults
pickleshare               0.5                      py35_0    defaults
pip                       8.1.1                    py35_1    defaults
proj.4                    4.9.1                         2    conda-forge
ptyprocess                0.5                      py35_0    defaults
pygments                  2.1.1                    py35_0    defaults
pyparsing                 2.0.3                    py35_0    defaults
pyqt                      4.11.4                   py35_1    defaults
python                    3.5.1                         0    defaults
python-dateutil           2.5.2                    py35_0    defaults
python.app                1.2                      py35_4    defaults
pytz                      2016.3                   py35_0    defaults
pyzmq                     15.2.0                   py35_0    defaults
qt                        4.8.7                         1    defaults
qtconsole                 4.2.1                    py35_0    defaults
rasterio                  0.32.0              np110py35_0    conda-forge
readline                  6.2                           2    defaults
scipy                     0.17.0              np110py35_0    defaults
setuptools                20.3                     py35_0    defaults
simplegeneric             0.8.1                    py35_0    defaults
sip                       4.16.9                   py35_0    defaults
six                       1.10.0                   py35_0    defaults
snuggs                    1.3.1               np110py35_0    defaults
sqlite                    3.9.2                         0    defaults
terminado                 0.5                      py35_1    defaults
tk                        8.5.18                        0    defaults
tornado                   4.3                      py35_0    defaults
traitlets                 4.2.1                    py35_0    defaults
wheel                     0.29.0                   py35_0    defaults
xerces-c                  3.1.2                         0    defaults
xz                        5.0.5                         1    defaults
zlib                      1.2.8                         0    defaults

[jakirkham]

It looks like you are not getting our version hdf5, but an older one from the defaults channel.

[ocefpaf]

It looks like you are not getting our version hdf5, but an older one from the defaults channel.

Yes. My I suspect that rasterio is pulling libnetcdf b/c of our gdal. That means we need to build libnetcdf with our hdf5 to prevent this.

@mhearne-usgs does the workaround in #5 (comment) work for you?

[jakirkham]

Ah, thanks for point this out. Yeah, these channel fixes will be nice once settled on and released.

[mhearne-usgs]

#5 is working so far... I'm going to test with the rest of my dependencies and see what else happens. I appreciate everyone being so responsive with these fixes and workarounds!

[ocefpaf]

As soon as we sort conda-forge/staged-recipes#213 out we won't need that workaround.

[ocefpaf]

@jakirkham I believe we can close this since you merged #7

[jakirkham]

Was basically leaving this open until I heard an all good. Also, it looked like there were some upstream issues we were tracking. If the pinning is working nicely for everyone, will go ahead and close. If issues arise, feel free to reopen or ping me and I can reopen it. Thanks everyone for being so patient as we sort this out.

[ocefpaf]

The only thing we must pay attention is to unpin hdf5 (or pin to the latest version) once we finish libnetcdf.

[jakirkham]

Yeah, please remind me if I forget. Though I think there is quite bit more HDF5 unpinning needed throughout the stack.

[ocefpaf]

Though I think there is quite bit more HDF5 unpinning needed throughout the stack.

Lots of 😄

[jakirkham]

...or after this experience repinning. I'm having a cathartic moment where I am learning to stop worrying and love the pinning (Dr. Strangelove anyone). 😄

In all seriousness though. With proper automation and build number increments, it starts to behave like one repo of recipes even though we have many repos. It also sidesteps much of the pain of ABI incompatibility or a proper way of addressing this in conda-build recipes.

[ocefpaf]

...or after this experience repinning. I'm having a cathartic moment
where I am learning to stop worrying and love the pinning (Dr. Strangelove
anyone).

Love the reference.

In all seriousness though. With proper automation and build number
increments, it starts to behave like one repo of recipes even though we
have many repos. It also sidesteps much of the pain of ABI incompatibility
or a proper way of addressing this in conda-build recipes.

That's the plan!