Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing mixed Rust/Python project fails when structured as recommended #490

Closed
ArniDagur opened this issue Apr 7, 2021 · 28 comments · Fixed by frank1010111/pywaterflood#62

Comments

@ArniDagur
Copy link

The README of this project says the following:

Mixed rust/python projects

To create a mixed rust/python project, create a folder with your module name (i.e. lib.name in Cargo.toml) next to your Cargo.toml and add your python sources there:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   └── bar.py
├── Readme.md
└── src
    └── lib.rs

maturin will add the native extension as a module in your python folder. When using develop, maturin will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.

With cffi you can do from .my_project import lib and then use lib.my_native_function, with pyo3/rust-cpython you can directly from .my_project import my_native_function.

Example layout with pyo3 after maturin develop:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   ├── bar.py
│   └── my_project.cpython-36m-x86_64-linux-gnu.so
├── Readme.md
└── src
    └── lib.rs

This is the structure I'm using for python-adblock. The project is written entirely in Rust, but I'm using a mixed Rust/Python project in order to include Python type stubs. I have an __init__.py which imports the neccesary symbols from the native module.

However, some of my users have reported that the imports inside __init__.py fail as follows:

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from .adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

or

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from adblock.adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

See ArniDagur/python-adblock#17. As you can see, the result is the same whether I use from .adblock import ... or from adblock.adblock import .... I have only managed to reproduce this when the package is installed system wide (pip3 install .) and the current working directory is the root of my project (git repository).

Does anyone know what is going on? Am I doing something different from what is described in the README.md, or is the advice in the README.md wrong in some way?

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@konstin

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@messense
Copy link
Member

messense commented May 13, 2021

pip isn't used, but I've added python3 -m sysconfig for these archs.

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391397

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391398

armv7: Platform: "linux-armv8l"
armhf: Platform: "linux-armv8l"

Interesting. I think we are expecting Platform: "linux-armv7l"

I wonder what's is Rust target triple on these platforms.

See also randombit/botan#1543 (comment)

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@messense

This comment was marked as resolved.

@omnivagant

This comment was marked as resolved.

messense referenced this issue in tikv/client-py Sep 1, 2021
Signed-off-by: Andy Lok <andylokandy@hotmail.com>
@stusmall
Copy link

stusmall commented Mar 4, 2022

I'm running into this same issue, but I might have a shorter path to reproducing it. I'm including as much detail as possible because I might be doing something weird in here. I'm not an experienced python dev and I'm kinda winging it as I go.

  1. Check out maturin @ revision d531e49 (head of main when I pulled)
  2. Install with cargo install -f --path .
  3. Goto the cffi-mixed test crate with cd test-crates/cffi-mixed
  4. Create and activate a venv. python3 -m venv venv; source venv/bin/activate
  5. Install tox pip install tox
  6. Run tox tox --skip-missing-interpreters (I'm just using the system interpreter to simply things):
(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
.package create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.package
.package installdeps: maturin>=0.12,<0.13
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38
py38 installdeps: pytest
py38 inst: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='1058416854'
py38 run-test: commands[0] | pytest
================================= test session starts ==================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 0 items / 1 error

======================================== ERRORS ========================================
______________ ERROR collecting test-crates/cffi-mixed/test_cffi_mixed.py ______________
ImportError while importing test module '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/test_cffi_mixed.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../../.pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_cffi_mixed.py:3: in <module>
    import cffi_mixed
cffi_mixed/__init__.py:1: in <module>
    from .cffi_mixed import ffi, lib
E   ModuleNotFoundError: No module named 'cffi_mixed.cffi_mixed'
=============================== short test summary info ================================
ERROR test_cffi_mixed.py
!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!
=================================== 1 error in 0.08s ===================================
ERROR: InvocationError for command /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38/bin/pytest (exited with code 2)
_______________________________________ summary ________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
ERROR:   py38: commands failed

For some extra version info just in case it helps for reproducing the issue:

(venv) ➜  cffi-mixed git:(main) python3 --version
Python 3.8.12
(venv) ➜  cffi-mixed git:(main) pip --version
pip 21.1.1 from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv) ➜  cffi-mixed git:(main) tox --version
3.24.5 imported from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/tox/__init__.py

What is interesting is that if I run maturin develop before running tox --skip-missing-interpreters then the test works.

env) ➜  cffi-mixed git:(main) maturin develop
🍹 Building a mixed python/rust project
🐍 Using CPython 3.8 at /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python to generate the cffi bindings
📡 Using build options bindings from pyproject.toml
Collecting cffi
  Using cached cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.15.0 pycparser-2.21
WARNING: You are using pip version 21.1.1; however, version 22.0.3 is available.
You should consider upgrading via the '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python -m pip install --upgrade pip' command.
   Compiling cffi-mixed v0.1.0 (/home/stusmall/Workspace/maturin/test-crates/cffi-mixed)
    Finished dev [unoptimized + debuginfo] target(s) in 0.55s
📦 Built wheel to /tmp/.tmpsizYQW/cffi_mixed-0.1.0-py3-none-linux_x86_64.whl
🛠  Installed cffi-mixed-0.1.0
(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 inst-nodeps: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='2429669310'
py38 run-test: commands[0] | pytest
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 3 items

test_cffi_mixed.py ...                                                                                                                                                                                        [100%]

================================================================================================= 3 passed in 0.02s =================================================================================================
______________________________________________________________________________________________________ summary ______________________________________________________________________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
  py38: commands succeeded
  congratulations :)

So before I go further, I want to repeat, I'm not a python dev. So this might be missing the point of the initial bug or misunderstanding how modules work. But it looks to me that when from .cffi_mixed import ffi, lib is run instead of importing from cffi_mixed the wheel it is opening the directory on disk cff_mixed. When this happens before maturin develop is run, it doesn't have native.so in place, but it does afterwards and so the tests pass

@messense
Copy link
Member

@stusmall Sorry for the late reply, I think your hypothesis is correct. We've changed the default project template to include a python folder in #855 to avoid such issue.

vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
vlaci added a commit to onekey-sec/unblob-native that referenced this issue May 14, 2023
It is needed to prevent import errors as the repo root sometimes gets
into the PYTHONPATH, e.g. when python interactive is used. In that
case the native extension may not be loadable.

See also PyO3/maturin#490
SemyonSinchenko added a commit to SemyonSinchenko/farsante that referenced this issue Mar 10, 2024
- replace poetry by maturin
- introduce Makefile for simplify life of devs
- add a maturin action
- refactor the project to the recommended structure
- redefine dependencies as pure pyproject

(see PyO3/maturin#490 for details)

 On branch 18-maturin_build
 Changes to be committed:
	new file:   .github/workflows/release.yaml
	modified:   .gitignore
	renamed:    h2o-data-rust/Cargo.lock -> Cargo.lock
	renamed:    h2o-data-rust/Cargo.toml -> Cargo.toml
	new file:   Makefile
	deleted:    h2o-data-rust/README.md
	modified:   poetry.lock
	modified:   pyproject.toml
	renamed:    farsante/__init__.py -> python/farsante/__init__.py
	renamed:    farsante/h2o_dataset_create.py -> python/farsante/h2o_dataset_create.py
	renamed:    farsante/h2o_dataset_create_all.py -> python/farsante/h2o_dataset_create_all.py
	renamed:    farsante/pandas_dfs.py -> python/farsante/pandas_dfs.py
	renamed:    farsante/pandas_generators.py -> python/farsante/pandas_generators.py
	renamed:    farsante/pyspark_dfs.py -> python/farsante/pyspark_dfs.py
	renamed:    farsante/pyspark_generators.py -> python/farsante/pyspark_generators.py
	renamed:    h2o-data-rust/src/generators.rs -> src/generators.rs
	renamed:    h2o-data-rust/src/helpers.rs -> src/helpers.rs
	renamed:    h2o-data-rust/src/lib.rs -> src/lib.rs
	renamed:    h2o-data-rust/src/main.rs -> src/main.rs
SemyonSinchenko added a commit to MrPowers/farsante that referenced this issue Mar 10, 2024
* Building system of farsante

- replace poetry by maturin
- introduce Makefile for simplify life of devs
- add a maturin action
- refactor the project to the recommended structure
- redefine dependencies as pure pyproject

(see PyO3/maturin#490 for details)

 On branch 18-maturin_build
 Changes to be committed:
	new file:   .github/workflows/release.yaml
	modified:   .gitignore
	renamed:    h2o-data-rust/Cargo.lock -> Cargo.lock
	renamed:    h2o-data-rust/Cargo.toml -> Cargo.toml
	new file:   Makefile
	deleted:    h2o-data-rust/README.md
	modified:   poetry.lock
	modified:   pyproject.toml
	renamed:    farsante/__init__.py -> python/farsante/__init__.py
	renamed:    farsante/h2o_dataset_create.py -> python/farsante/h2o_dataset_create.py
	renamed:    farsante/h2o_dataset_create_all.py -> python/farsante/h2o_dataset_create_all.py
	renamed:    farsante/pandas_dfs.py -> python/farsante/pandas_dfs.py
	renamed:    farsante/pandas_generators.py -> python/farsante/pandas_generators.py
	renamed:    farsante/pyspark_dfs.py -> python/farsante/pyspark_dfs.py
	renamed:    farsante/pyspark_generators.py -> python/farsante/pyspark_generators.py
	renamed:    h2o-data-rust/src/generators.rs -> src/generators.rs
	renamed:    h2o-data-rust/src/helpers.rs -> src/helpers.rs
	renamed:    h2o-data-rust/src/lib.rs -> src/lib.rs
	renamed:    h2o-data-rust/src/main.rs -> src/main.rs

* Drop poetry.lock

By switching to maturing-builds
we do not need poetry.lock anymore

 On branch 18-maturin_build
 Your branch is up to date with 'origin/18-maturin_build'.

 Changes to be committed:
	deleted:    poetry.lock
Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue May 14, 2024
… projects

The previous layout leads to an import error when installing with `maturin build` and `pip install .`.

This error was common enough that `maturin` changed the recommended project layout to what this commit does.

A prior PR attempted to solve this by altering `lib.name` in Cargo.toml, but that did not work for me.

- [Prior PR](apache#694)
- [maturin ImportError issue](PyO3/maturin#490)
- [maturin changes recommended project structure](PyO3/maturin#855)
andygrove pushed a commit to apache/datafusion-python that referenced this issue May 14, 2024
#695)

* chore: update to maturin's recommended project layout for rust/python projects

The previous layout leads to an import error when installing with `maturin build` and `pip install .`.

This error was common enough that `maturin` changed the recommended project layout to what this commit does.

A prior PR attempted to solve this by altering `lib.name` in Cargo.toml, but that did not work for me.

- [Prior PR](#694)
- [maturin ImportError issue](PyO3/maturin#490)
- [maturin changes recommended project structure](PyO3/maturin#855)

* ci: update `ruff check` for nested python directory
hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
hohav added a commit to hohav/peppi-py that referenced this issue Jul 8, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
hohav added a commit to hohav/peppi-py that referenced this issue Jul 15, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
hohav added a commit to hohav/peppi-py that referenced this issue Aug 1, 2024
Per PyO3/maturin#490 and
PyO3/maturin#855, this avoids problems when
importing from the project dir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants