[AIRFLOW-XXXX] Adjust celery defaults to work with breeze #7205

turbaszek · 2020-01-18T14:57:37Z

This PR changes default Celery configs values to ease working with CeleryExecutor in Breeze.

Issue link: Document only change, no JIRA issue

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

turbaszek · 2020-01-18T14:58:39Z

airflow/config_templates/default_celery.py

@@ -40,7 +40,7 @@ def _broker_supports_visibility_timeout(url):
        broker_transport_options['visibility_timeout'] = 21600

 DEFAULT_CELERY_CONFIG = {
-    'accept_content': ['json', 'pickle'],
+    'accept_content': ['json'],


Reason for this changes:

Running a worker with superuser privileges when the worker accepts messages serialized with pickle is a very bad idea! If you really want to continue then you have to set the C_FORCE_ROOT environment variable (but please think about this before you do). User information: uid=0 euid=0 gid=0 egid=0

codecov-io · 2020-01-18T15:40:11Z

Codecov Report

Merging #7205 into master will decrease coverage by 0.25%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #7205      +/-   ##
==========================================
- Coverage   85.37%   85.12%   -0.26%     
==========================================
  Files         723      753      +30     
  Lines       39543    39709     +166     
==========================================
+ Hits        33761    33801      +40     
- Misses       5782     5908     +126

Impacted Files	Coverage Δ
airflow/config_templates/default_celery.py	`52.94% <ø> (ø)`	⬆️
...rflow/contrib/sensors/sagemaker_training_sensor.py	`0% <0%> (-100%)`	⬇️
airflow/kubernetes/volume_mount.py	`44.44% <0%> (-55.56%)`	⬇️
airflow/kubernetes/volume.py	`52.94% <0%> (-47.06%)`	⬇️
airflow/kubernetes/pod_launcher.py	`45.25% <0%> (-46.72%)`	⬇️
airflow/kubernetes/refresh_config.py	`50.98% <0%> (-23.53%)`	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	`76.47% <0%> (-22.33%)`	⬇️
airflow/jobs/backfill_job.py	`90.43% <0%> (-1.45%)`	⬇️
airflow/cli/commands/dag_command.py	`85.9% <0%> (-0.31%)`	⬇️
airflow/jobs/scheduler_job.py	`89.23% <0%> (-0.12%)`	⬇️
... and 63 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 417feda...e91ecbe. Read the comment docs.

(cherry picked from commit 24ca556)

)" This reverts commit 5f8f00d.

(cherry picked from commit 24ca556)

)" This reverts commit 5f8f00d.

(cherry picked from commit 24ca556)

)" This reverts commit 5f8f00d.

(cherry picked from commit 24ca556)

)" This reverts commit 5f8f00d.

Part of apache#7205 (cherry picked from commit b65ed471a662ee4d90420a90d093b27ceeb4a227)

Part of apache#7205

Part of apache#7205 (cherry picked from commit b65ed471a662ee4d90420a90d093b27ceeb4a227)

Part of #7205

I think this change was unintentional -- apache#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense.

This change was unintentional -- #7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense.

This change was unintentional -- #7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. (cherry picked from commit 6c458f2)

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

Part of apache#7205

This change was unintentional -- apache#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. (cherry picked from commit 6c458f2) (cherry picked from commit 6f8f23a)

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

This change was unintentional -- apache/airflow#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. (cherry picked from commit 6c458f29c0eeadb1282e524e76fdd379d6436824) GitOrigin-RevId: 6f8f23a019f61580d4fe46aa0a7c196478cef6d0

This change was unintentional -- apache/airflow#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. GitOrigin-RevId: 6c458f29c0eeadb1282e524e76fdd379d6436824

This change was unintentional -- apache/airflow#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. (cherry picked from commit 6c458f29c0eeadb1282e524e76fdd379d6436824) GitOrigin-RevId: 6f8f23a019f61580d4fe46aa0a7c196478cef6d0

This change was unintentional -- apache/airflow#7205 That PR just changed it to work with breeze. Since we had `16` as default in 1.10.x and to get better performance and keep in line with `dag_concurrency` and `max_active_runs_per_dag` -- I think `16` makes more sense. GitOrigin-RevId: 6c458f29c0eeadb1282e524e76fdd379d6436824

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze

e91ecbe

turbaszek commented Jan 18, 2020

View reviewed changes

turbaszek requested a review from potiuk January 18, 2020 14:59

potiuk approved these changes Jan 18, 2020

View reviewed changes

potiuk merged commit 24ca556 into apache:master Jan 18, 2020

potiuk pushed a commit that referenced this pull request Jan 21, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205)

931525a

(cherry picked from commit 24ca556)

kaxil pushed a commit that referenced this pull request Jan 22, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205)

5f8f00d

(cherry picked from commit 24ca556)

kaxil added a commit that referenced this pull request Jan 23, 2020

Revert "[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205

4ac97fd

)" This reverts commit 5f8f00d.

kaxil pushed a commit that referenced this pull request Jan 23, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205)

01e9f53

(cherry picked from commit 24ca556)

kaxil added a commit that referenced this pull request Jan 23, 2020

Revert "[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205

d5ffa07

)" This reverts commit 5f8f00d.

potiuk pushed a commit that referenced this pull request Jan 26, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205)

53810d8

(cherry picked from commit 24ca556)

potiuk pushed a commit that referenced this pull request Jan 26, 2020

Revert "[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205

c32b7e5

)" This reverts commit 5f8f00d.

kaxil pushed a commit that referenced this pull request Jan 26, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205)

8e2109b

(cherry picked from commit 24ca556)

kaxil added a commit that referenced this pull request Jan 26, 2020

Revert "[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (#7205

511abcc

)" This reverts commit 5f8f00d.

galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze (apache#7205)

765482b

turbaszek deleted the AIRFLOW-XXXX-celery-defaults branch March 14, 2020 23:35

kaxil added a commit to astronomer/airflow that referenced this pull request Jun 15, 2020

Fix Celery default to no longer allow pickle (apache#7205)

d819da6

Part of apache#7205 (cherry picked from commit b65ed471a662ee4d90420a90d093b27ceeb4a227)

kaxil added a commit to astronomer/airflow that referenced this pull request Jun 15, 2020

Fix Celery default to no longer allow pickle (apache#7205)

45a2a33

Part of apache#7205

kaxil added a commit to astronomer/airflow that referenced this pull request Jun 15, 2020

Fix Celery default to no longer allow pickle (apache#7205)

8a3076b

Part of apache#7205 (cherry picked from commit b65ed471a662ee4d90420a90d093b27ceeb4a227)

kaxil added a commit that referenced this pull request Jun 16, 2020

Fix Celery default to no longer allow pickle (#7205)

0054a20

Part of #7205

potiuk pushed a commit that referenced this pull request Jun 29, 2020

Fix Celery default to no longer allow pickle (#7205)

967e61d

Part of #7205

kaxil added a commit that referenced this pull request Jul 1, 2020

Fix Celery default to no longer allow pickle (#7205)

7f20bed

Part of #7205

kaxil mentioned this pull request Jan 11, 2021

Change the default celery worker_concurrency to 16 #13612

Merged

leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Feb 11, 2021

Fix Celery default to no longer allow pickle (#7205)

5e2c3c1

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Feb 23, 2021

Fix Celery default to no longer allow pickle (#7205)

d831d24

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021

Fix Celery default to no longer allow pickle (apache#7205)

94eec9c

Part of apache#7205

leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 3, 2021

Fix Celery default to no longer allow pickle (#7205)

4ddda43

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 4, 2021

Fix Celery default to no longer allow pickle (#7205)

0ef3660

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 4, 2021

Fix Celery default to no longer allow pickle (#7205)

eba8b4e

Part of apache/airflow#7205 GitOrigin-RevId: 7f20bedc3aaece73acb8b741c84720da66336ed4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze #7205

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze #7205

turbaszek commented Jan 18, 2020 •

edited

Loading

turbaszek Jan 18, 2020

codecov-io commented Jan 18, 2020

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze #7205

[AIRFLOW-XXXX] Adjust celery defaults to work with breeze #7205

Conversation

turbaszek commented Jan 18, 2020 • edited Loading

turbaszek Jan 18, 2020

Choose a reason for hiding this comment

codecov-io commented Jan 18, 2020

Codecov Report

turbaszek commented Jan 18, 2020 •

edited

Loading