-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow PythonVenvOperator using other index url #33017
Allow PythonVenvOperator using other index url #33017
Conversation
I would prefer the changes are split into separate pull requests. The repository uses squash-merge and the separate commits would be lost in Git history. |
1c20698
to
851a23c
Compare
Good idea @uranusjr , did not think about the squash. Splitted the ground work into #33051 - This PR branch is based on the other PR's branch (now). Change are only the two latter commits, the first both will go away if the other is merged. |
@uranusjr Now the spin-off from the base rework is merged, this PR is reduced to the core feature of having a "Package Index" core hook/connection and the option to use this in VirtualEnv operators |
5a1de9d
to
80015e3
Compare
One minor comment, overall looks good to me. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks really goood. But I have a question about separating out BranchPythonVirtualenvOperator to a separate PR.
* Extended PythonVirtualEnvOperator for extra index URL * Separate-out BranchPythonVirtualenvOperator from this PR
This PR adds the option to specify extra index URLs to PythonVirtualEnvOperator (+corresponding decorator) in order to be able to install virtualenvs with (private) additional Python package repositories.
Besides the option to modify the worker config or adding ENVs to specify additional PIP install configs, this allows to use different back-ends per DAG/Task.
As the definition of (private) extra index URLs might require credentials to be passed, using plain requirements or pip install options would expose passwords as secrets in logs. Therefore the Airflow core provided connection types were extended to contain a "Package Index (Python)" connection type that can be used to store the credentials in Airflow in a (more) secret way and preventing to expose.
Summary of changes, separated by commit:
Note: The PR was based on a discussion in Slack in https://apache-airflow.slack.com/archives/CCPRP7943/p1684608779378759
Note (2): Caching of venvs was separated in a follow-up PR boschglobal#2
How and what to test:
@AutomationDev85 @clellmann @wolfdn