Building Rust code and shipping binary wheels

[ghost]

Originally reported by: SimonSapin (Bitbucket: SimonSapin, GitHub: SimonSapin)

---

This is an assistance request more than a bug report, please let me know if this is not the place to ask.

Rust libraries can provide a C-compatible API that can be used from C or other languages that can call into C like Python. Cargo produces a dynamic/shared library (e.g. a .so file on Linux), and that file can be used with ctypes or CFFI.

This works, but I’d like to package it all in a Python library that integrates with setuptools so that users can pip install it without thinking about Rust. Installing from source would build the Rust code automatically as long as cargo and rustc are somewhere in the user’s PATH. Ideally, I’d also ship platform-specific binary wheels that contain the compiled .so file so that most users don’t even need to have a Rust compiler.

I think I mostly have the first part so far, using a distutils.setup_keywords entry point (inspired by CFFI):

• setup.py
• setuptools_ext

pip install -e . (which uses python setup.py develop) and tox (which uses python setup.py sdist and pip install with the generated .zip file) seem to work, but pip install . (which in recent versions uses python setup.py bdist_wheel) doesn’t: the .whl file (when opened as a ZIP) does not contain the .so file.

I’d appreciate any suggestion on:

• What is the right place to hook into setuptools for doing this kind of thing? I’ve also tried overriding the build_ext command instead of distutils.setup_keywords.
• How can I convince bdist_wheel to ship a .so file so that it’s available at run time once the package is installed?
• Bonus: I’ve managed to make the suffix change from py3-none-any.whl to cp34-cp34m-linux_x86_64.whl by monkey-patching Distribution.is_pure to return false. Can I make it something like any-none-linux_x86_64.whl to make wheels platform-specific (for the binary .so) but independent of the Python version or implementation?

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/435

[ghost]

Original comment by SimonSapin (Bitbucket: SimonSapin, GitHub: SimonSapin):

---

From reading source code, it looks like bdist_wheel does a lot of interesting things in the presence of a distutils.extension.Extension, but that seems to be deeply tied to a C compiler where distutils controls the build end-to-end. What I’d like is "here is an arbitrary command that you can run to produce a native .so file".

[ghost]

Original comment by kmike (Bitbucket: kmike, GitHub: kmike):

---

I have nothing to say from technical point of view, but this feature should be very helpful; I wanted to do exactly the same, but haven't started the investigation. There is some Rust-based software (like rust-url) with no good C/C++ alternatives I'm aware of.

[dholth]

Not seeing a lot of interesting things happening in bdist_wheel for Extension, just is_pure = False and get_tag(). We already have a command line parameter for the python tag (py3 or py2.py3 would probably do for you) and an upcoming tag for abi3 (works on CPython 3.2 and above with a single extension). It would be pretty easy to add another option for "dlopen-style" tags and return that tag in bdist_wheel's get_tag function. You could put the parameter in setup.cfg.

You probably have to add your Cargo-generated .so to the list of files to be installed in a different way, using a different corner of setuptools, to get it into the wheel. Maybe package_data?

[fafhrd91]

just pushed https://github.com/fafhrd91/setuptools-rust
you can use bdist_egg and bdist_wheel with it, but it only for extensions

[webknjaz]

I believe it's also possible to achieve this with https://github.com/getsentry/milksnake