py_library imports ignored when linking into pex_binary

[evanj]

TL;DR: If I have a py_library(..., srcs=["foo.py"], imports=["."]), then with the standard Bazel Python rules I can use import foo to reference it. If I link this py_library into a pex_binary, it ignores the imports part, so the files get packed into the pex as full/workspace/path/foo.py, instead of as foo.py in the root of the pex.

Expected result: I should be able to change standard Bazel py_binary into a pex_binary and have it magically work. This probably means shuffling files around, or packaging each py_library into a wheel, and packaging those together into the pex?

Additional details

I have a Python mono-repo that is set up something like this:

• package_foo/setup.py - A "standard" setuptools setup.py to build a wheel
• package_foo/foo/__init__.py - Python package foo
• somebinary/bin.py - A Python program that uses import foo

(Imagine more packages and binaries)

Each subdirectory is a "standard" Python package. We are still evaluating the various monorepo build tools (Pants, Buck, Bazel), so this lets us make it work fairly easily. I'd like to just "drop in" the appropriate BUILD rules, but I can't figure out how to make pex_binary work in this case, since I need to "fix" the import path somehow.

PS. Thanks for creating these rules! Being able to directly link wheels is great.

[evanj]

Turns out this is currently impossible due to bazelbuild/bazel#2617 ; Bazel does not expose the imports attribute. As a result, it is impossible to write Skylark rules to do this.

[benley]

Aye, it's true. Sorry I forgot to say as much when you opened this issue, because I spent a while figuring it out last fall. Let's keep this issue open because I'm pretty sure bazelbuild/bazel#2791 will be merged eventually, and then it can actually work :-)

[benley]

#35 should fix this when bazel's support is ready

[evanj]

In the meantime, since this still hasn't landed, I've implemented a pythonroot attribute that lets me say "subdirectory //other_python_root" is a "root" of the Python import tree. I then rewrite the pex manifest to make that work. If there is interest: I'd be happy to contribute a patch back to this project.

[c4urself]

@evanj -- I'd definitely be interested in a patch but looks like it'd be similar to (#42)? I have a similar project (large-ish python monorepo) that I'd like to use Bazel with. Would be really interested in hearing your findings compared to Buck/Pants.

[evanj]

Yes, it is very similar, although my version adds a pythonroot attribute to pex_library to be able to move any code around in the bazel workspace, without consuming tests/binaries needing to be aware.

As far as using Bazel entirely: The jury is still out, but so far: we've been able to get most of our stuff working. It definitely is reliable: if you can get it to work on your machine, it seems to work on everyone's machine, which is really the issue we wanted to solve. It also allows us to easily express complex dependencies between parts of the code base (which we also wanted to enable). The one caveat: if you consume very large third-party libraries (e.g. pandas, which depends on scipy and numpy), rebuilding any pex rules is really slow. I'm testing a hack to make this slightly better.

My take: Bazel's Python support is pretty bad at the moment, but I see a heck of a lot more "momentum" behind Bazel (eg. Dropbox, Stripe, others) so we are giving it a shot.

[c4urself]

@evanj would you be able to share your patch?

[evanj]

@c4urself Sorry for the delay. See the branch at https://github.com/evanj/bazel_rules_pex/tree/pythonroot . I did not send a pull request for this, since it isn't totally clear to me if this is the "right" design for this.