PIP dependency support #1

mattmoor · 2017-09-05T22:36:31Z

This is a proposal for some PIP rules that are heavily based on a prototype I wrote a month or two back. Very open to suggestions here, but it seems to work across a number of examples I've tried so far.

This is broken off of my prototype repository, and there are a handful of TODOs left to resolve.

Elaborate on why the TODO to rewrite {pip,whl}.sh as Python is harder than it looks.

damienmg

No big remarks, except adding the bazel ci configuration.

damienmg · 2017-09-06T07:56:49Z

.gitignore

@@ -0,0 +1,39 @@
+# Compiled Object files


Why have all those parts? emacs exclude belongs to the ~/.gitignore, all the bin files should not be generated by bazel

This is just my standard .gitignore file I copy from repo to repo. I never touch it again.

Repo .gitignore are supposed to be repo specific, so not includes things like *~...

I advice I got, didn't like, but eventually learned to appreciate the wisdom of, is to create a ~/.gitignore_global with all the editor autosave filenames, etc (things that are particular to me, not to the repo I'm working on), and do git config --global core.excludesfile ~/.gitignore_global

Otherwise, you're like "What part of this Python program creates Fortran executables?!"

damienmg · 2017-09-06T07:59:42Z

.travis.yml

@@ -0,0 +1,27 @@
+sudo: required


Add .ci/rules_python.json for CI. see https://github.com/bazelbuild/continuous-integration/blob/master/docs/owner.md#customizing-a-project

Seeing the build, you probably just want:

[ { "variation": "", "configurations": [ {"node": "linux-x86_64"}, {"node": "ubuntu_16.04-x86_64"}, {"node": "darwin-x86_64"} ] } ]

damienmg · 2017-09-06T08:02:28Z

README.md

+which can be used directly in `deps=[]` to reference an imported `py_library`.
+
+```python
+load("@my_deps//:requirements.txt", "packages")


You probably meant requirements.bzl here

Not updated? Maybe you just want to delete that part all in profit of the skydoc generated.

Ha, I think this bug was in two places. thanks for catching it again :)

damienmg · 2017-09-06T08:04:13Z

python/pip.sh

@@ -0,0 +1,84 @@
+#!/bin/bash -e


General comment for this file: the file already complex, why not in python instead?

See line 17. I was halfway through rewriting one of these scripts in Python before I realized: I can't invoke a py_binary during WORKSPACE, I'd have to publish a PAR file. I immediately git stashed and rewrote my TODO instead. :(

:( I actually do things without py_binary for simple python_script (I know that is a shame) using simply python <script.py>. Anyway up to you.

I suppose that is true. I want to be able to factor libraries and have a clean separation of unit tests etc. I'll experiment with my stashed changes with this in mind.

I would say it's ok to use shell here, not Python. But you actually need Python inside the script to parse json, so yeah.

For historical reasons that are still sometimes relevant, don't put the -e in the shebang line, instead add set -euo pipefail around line 16

The only remaining shell script now does this (I copied it from you :D)

damienmg · 2017-09-06T08:05:50Z

python/whl.sh

@@ -0,0 +1,82 @@
+#!/bin/bash -e
+


Same here, bash can become quickly hard to read...

mattmoor · 2017-09-06T17:16:54Z

@duggelz Do you have any suggestions on the best way to download and invoke pip without using get-pip.py to install it as a system-wide dependency? This seems to be why the OSX build is failing.

mattmoor · 2017-09-06T23:02:56Z

@sebgoa You'd asked for this at one point, if you are still interested I'd love your feedback.

duggelz · 2017-09-08T01:11:43Z

README.md

+See Bazel core [documentation](https://docs.bazel.build/versions/master/be/python.html#py_test).
+
+<a name="pip_import"></a>
+## pip_import


There's a thing called skydoc that will turn structured docstrings into Bazel documentation. However, it's kind of basic and a bit, well, not beautiful. https://google.github.io/subpar/subpar.html

I'll TAL at skydoc for this repo, but in rules_docker and rules_k8s I'm blocked on this

Ok, I've switched over to skydoc for the docs here (based on how google/subpar does it). Feedback welcome :)

mattmoor · 2017-09-08T17:35:03Z

I moved whl.sh into a little Python script that I can invoke directly, but also reasonably unit test.

I have not moved pip.sh yet, and thinking about it left me with an idea. Considering:

Shelling out to pip from Python is a bit silly,
@damienmg's trick won't work if I have a pip library dependency,
I still haven't solved the pip fetching problem.

What if we build / publish a PAR (similar to the process rules_docker uses), which links the pip library, and removes the need to get-pip.py somehow? WDYT?

I still haven't found a clean process for doing this from a single repo (google/containerregistry and bazelbuild/rules_docker are separate repos), but that seems like a smaller problem to solve.

mattmoor · 2017-09-10T16:30:30Z

I found this, which may be relevant to why certs aren't working properly from pip in a PAR file. It is similar to the trick we do for httplib2 in google/containerregistry tools.

This migrates the logic from pip.sh into piptool.py, which should improve portability by removing the bash dependency. This also has the beginnings of wrapping piptool as a closed redistributable that doesn't rely on a system-installed copy of PIP, but instead uses these rules to pull pip into a PAR bundle. Besides needing to work out the details of releasing and redistributing the PAR, we have two unresolved issues: * When bundled as a PAR (vs. py_binary), piptool seems to pick up the system-installed version of pip. * When bundled as a PAR, piptool sometimes sees cert issues resolving requirements (similar to what we see with httplib2).

duggelz · 2017-09-12T21:28:38Z

examples/version/version_test.py

+
+ def test_version(self):
+ parts = pip.__version__.split('.')
+ self.assertEqual(3, len(parts))


self.assertEqual(pip.__version__, '9.0.1')? You might be overthinking this :) There's no standard format for __version__, as you discovered, sometimes it's a tuple, sometimes a string, sometimes it doesn't exist.

duggelz · 2017-09-12T21:38:09Z

python/piptool.py

+# being invoked as a raw .py file. Once bundled, we should be able
+# to remove this fallback on a stub implementation of Wheel.
+try:
+ from python.whl import Wheel


I feel like you're really playing with fire calling your package python.

move python tools under a top-level rules_python package simplify version_test

mattmoor · 2017-09-12T23:32:31Z

@duggelz I think that's a very good question, and I think it breaks down where you have:

A and B have a_install() and b_install() functions that are also called in c/WORKSPACE

As you are clearly driving at, we have 3 sets of requirements that are being merged after the constraints have been solved, which I think is a losing proposition.

I think that my expectation is that instead of a_install and b_install we would pip_import a blended requirements.txt from A, B, C, and just abc_install.

This story is only half-baked because:

We don't have a rule for combining @A//:requirements.txt and @B//:requirements.txt (but I suspect this is pretty easy).
A, B, and C may each references this collection of dependencies with different repository names (but I believe this is doable by aliasing repositories with bind). At one point, with this exact problem in mind, I debated removing users capacity to name the repository itself, which would effectively guarantee a single resolution step, but I thought this was a bit extreme (e.g. what if I want to write an invoked tool that has its own requirements.txt, that's ok).

I think the crux of this issue is that WORKSPACE currently serves two roles (I hint at this above):

Resolving dependencies based on fuzzy constraints (incl. transitive resolution).
Fetching dependencies based on concrete specification.

The analogs to this outside of Bazel are things like [frozen] requirements.txt or package[-lock].json.

I think the guidance should be for users not to mix independently resolved external dependencies, but I'm not 100% sure if/how we can enforce it. Perhaps we can do this through Bazel's constraint system, but I have not looked any further at how we might express this.

Is what I'm describing clear? Does my logic make sense?

duggelz · 2017-09-15T01:42:46Z

@mattmoor I agree with what you wrote. Some additional points:

Bazel has a native.existing_rules that make fixing my points 1 and 2 easy.
Looking at various repository rules, there is a nice omit_foo pattern that allows C to override A and B as desired.
The conflict between "fuzzy constraints" and "concrete specification" is real. Python aims to address this with Pipfile vs Pipfile.lock, but meanwhile we muddle through.

I don't think we need to make a tool that tries to merge A and B automatically or anything. I think it's reasonable to say that C must provide a requirements.txt that satisfies every need that A and B have.

After reading some discussions about Java/Maven, I do think that we should put the names in a canonical format that we document and agree on. Specifically, I think we should have the literal prefix pypi_ on names that would otherwise be very short and ambigous. E.g. @pypi__package_name_here//:package_name or @pypi__package_name_here__version_here//:package_name

`whl_library` rules generated by `pip_import` are now named as: ``` pypi__{distribution}_{version} ``` Substituting illegal characters (e.g. `-`, `.`) with underscores.

mattmoor · 2017-09-15T14:43:37Z

@duggelz Given that things are hidden behind package(...) in the expected usage today, I think this is completely fine. The main change is that we wouldn't namespace packages based on the enclosing pip_import, which I think is fine.

I've gone with pypi__{distribution}_{version}, used existing_rules to avoid multiple imports, but each pip_import will still exposes the library through package() even if it didn't import it. I included a section in the README.md covering canonical naming.

Add a script for updating the tools and document this and the docs script.

mattmoor · 2017-09-15T18:20:58Z

@duggelz Hooray, green CI :)

Make pip_import more verbose

Update whl.bzl

This looks like it's left over from before the switch to buildkite. The Travis tests are not run in precommit now, and I can't even tell if they were run in PR bazelbuild#1. This causes pre-commit errors in forks if folks have Travis enabled in other projects.

This looks like it's left over from before the switch to buildkite. The Travis tests are not run in precommit now, and I can't even tell if they were run in PR #1. This causes pre-commit errors in forks if folks have Travis enabled in other projects.

…om-rules-pip

Add extra_index_urls parameter to pip_parse

mattmoor added 2 commits September 5, 2017 20:34

Check in an initial version of the pip rules.

13c47f3

This is broken off of my prototype repository, and there are a handful of TODOs left to resolve.

Remove some vestigial convenience code.

fa0d7b2

Elaborate on why the TODO to rewrite {pip,whl}.sh as Python is harder than it looks.

mattmoor requested review from duggelz and damienmg September 5, 2017 22:36

damienmg reviewed Sep 6, 2017

View reviewed changes

Incorporate Code Review Feedback

25c78ac

googlebot added the cla: yes label Sep 6, 2017

damienmg approved these changes Sep 6, 2017

View reviewed changes

duggelz reviewed Sep 8, 2017

View reviewed changes

mattmoor force-pushed the master branch 5 times, most recently from 76a4bcb to 91f1f9b Compare September 8, 2017 15:08

First cut at skydoc-based documentation.

1aafd84

mattmoor force-pushed the master branch from 91f1f9b to 1aafd84 Compare September 8, 2017 15:11

mattmoor force-pushed the master branch from 9b82191 to 116b99d Compare September 8, 2017 18:07

Move whl.sh into a Python script that has unit testing.

56be8e8

mattmoor force-pushed the master branch from 116b99d to 56be8e8 Compare September 8, 2017 18:25

pcj mentioned this pull request Sep 9, 2017

Python support requires pip or pypi pubref/rules_protobuf#3

Closed

mattmoor mentioned this pull request Sep 10, 2017

Feature request: pypi_package bazelbuild/bazel#699

Closed

mattmoor force-pushed the master branch 3 times, most recently from e2267eb to c1dbfa0 Compare September 10, 2017 15:37

mattmoor force-pushed the master branch from c1dbfa0 to ee457a0 Compare September 10, 2017 17:47

duggelz reviewed Sep 12, 2017

View reviewed changes

Incorporate review feedback

880a2fc

move python tools under a top-level rules_python package simplify version_test

mattmoor mentioned this pull request Sep 14, 2017

PAR picks up system-installed version instead of bundled version google/subpar#38

Closed

Adopt a canonical naming format for imported whl_library rules.

182554f

`whl_library` rules generated by `pip_import` are now named as: ``` pypi__{distribution}_{version} ``` Substituting illegal characters (e.g. `-`, `.`) with underscores.

mattmoor force-pushed the master branch 2 times, most recently from af2c63f to 182554f Compare September 15, 2017 17:54

Move the piptool dependency into a .PAR file.

a5d9f83

Add a script for updating the tools and document this and the docs script.

Fix buildifier issues.

66a1a92

mattmoor changed the title ~~[WIP] Proposal for PIP rules~~ PIP dependency support Sep 15, 2017

duggelz approved these changes Sep 15, 2017

View reviewed changes

mattmoor merged commit 7f4cc92 into bazelbuild:master Sep 15, 2017

This was referenced Sep 17, 2017

Defer pip related logic to python_rules tanin47/bazel-pex-pip#2

Open

For a script that depends on google-cloud re-builds are taking 70s benley/bazel_rules_pex#38

Open

duggelz mentioned this pull request Nov 10, 2017

PyYAML pip import fails with python3 #33

Closed

scele referenced this pull request in scele/rules_python Mar 12, 2018

Merge pull request #1 from scele/verbose_print

0d09a9a

Make pip_import more verbose

mdelaney pushed a commit to tubular/rules_python that referenced this pull request Mar 11, 2019

Merge pull request bazelbuild#1 from c4urself/patch-1

75c3e5f

Update whl.bzl

dhalperi mentioned this pull request Mar 2, 2020

Remove .travis.yml #284

Merged

alexeagle pushed a commit to alexeagle/rules_python that referenced this pull request Aug 19, 2020

Merge pull request bazelbuild#1 from dillon-giacoppo/dillon-import-fr…

2991357

…om-rules-pip

meastham referenced this pull request in tecton-ai/rules_python Aug 14, 2021

Merge pull request #1 from tecton-ai/extra_index_label

6eae336

Add extra_index_urls parameter to pip_parse

alexeagle mentioned this pull request Jan 3, 2023

Support gazelle plugin when using bzlmod #965

Closed

chrislovecnm mentioned this pull request Apr 6, 2023

gazelle modules_map is not running first #1157

Closed

This was referenced Jun 13, 2023

Update dependency rules_python to v0.34.0 uhthomas/adya#62

Open

chore(deps): update dependency rules_python to v0.34.0 uhthomas/automata#188

Open

adzenith mentioned this pull request Sep 26, 2023

Allow overriding the build rule gazelle uses for libraries/tests/binaries #1269

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PIP dependency support #1

PIP dependency support #1

mattmoor commented Sep 5, 2017

damienmg left a comment

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

damienmg Sep 6, 2017

duggelz Sep 8, 2017 •

edited

Loading

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

damienmg Sep 11, 2017

mattmoor Sep 11, 2017

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

duggelz Sep 12, 2017

duggelz Sep 12, 2017

mattmoor Sep 12, 2017

damienmg Sep 6, 2017

mattmoor Sep 6, 2017

mattmoor commented Sep 6, 2017

mattmoor commented Sep 6, 2017

duggelz Sep 8, 2017 •

edited

Loading

mattmoor Sep 8, 2017

mattmoor Sep 8, 2017

mattmoor commented Sep 8, 2017

mattmoor commented Sep 10, 2017

duggelz Sep 12, 2017

duggelz Sep 12, 2017

mattmoor commented Sep 12, 2017 •

edited

Loading

duggelz commented Sep 15, 2017

mattmoor commented Sep 15, 2017

mattmoor commented Sep 15, 2017

PIP dependency support #1

PIP dependency support #1

Conversation

mattmoor commented Sep 5, 2017

damienmg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duggelz Sep 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor commented Sep 6, 2017

mattmoor commented Sep 6, 2017

duggelz Sep 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor commented Sep 8, 2017

mattmoor commented Sep 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor commented Sep 12, 2017 • edited Loading

duggelz commented Sep 15, 2017

mattmoor commented Sep 15, 2017

mattmoor commented Sep 15, 2017

duggelz Sep 8, 2017 •

edited

Loading

duggelz Sep 8, 2017 •

edited

Loading

mattmoor commented Sep 12, 2017 •

edited

Loading