Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AsyncResolver to match ThreadedResolver behavior #8270

Merged
merged 46 commits into from
Apr 5, 2024
Merged

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Mar 30, 2024

What do these changes do?

AsyncResolver was disabled by default because it did not implement all of functionality of ThreadedResolver because aiodns did not support getaddrinfo until aio-libs/aiodns#118 which meant only IPv6 addresses were returned in most cases on systems that supported IPv6

see #559

Are there changes in behavior for the user?

AsyncResolver now behaves the same as the current default ThreadedResolver and will return both IPv6 and IPv4 addresses on systems that support it.

Is it a substantial burden for the maintainers to support this?

no

Related issue number

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt
    • The format is <Name> <Surname>.
    • Please keep alphabetical order, the file is sorted by names.
  • Add a new news fragment into the CHANGES/ folder
    • name it <issue_or_pr_num>.<type>.rst (e.g. 588.bugfix.rst)

    • if you don't have an issue number, change it to the pull request
      number after creating the PR

      • .bugfix: A bug fix for something the maintainers deemed an
        improper undesired behavior that got corrected to match
        pre-agreed expectations.
      • .feature: A new behavior, public APIs. That sort of stuff.
      • .deprecation: A declaration of future API removals and breaking
        changes in behavior.
      • .breaking: When something public is removed in a breaking way.
        Could be deprecated in an earlier release.
      • .doc: Notable updates to the documentation structure or build
        process.
      • .packaging: Notes for downstreams about unobvious side effects
        and tooling. Changes in the test invocation considerations and
        runtime assumptions.
      • .contrib: Stuff that affects the contributor experience. e.g.
        Running tests, building the docs, setting up the development
        environment.
      • .misc: Changes that are hard to assign to any of the above
        categories.
    • Make sure to use full sentences with correct case and punctuation,
      for example:

      Fixed issue with non-ascii contents in doctest text files
      -- by :user:`contributor-gh-handle`.

      Use the past tense or the present tense a non-imperative mood,
      referring to what's changed compared to the last released version
      of this project.

needs

@bdraco bdraco force-pushed the aiodns_fixes branch 2 times, most recently from 3db83a9 to b20c18a Compare March 30, 2024 01:41
AsyncResolver was disabled by default because it did not
implement all of functionality of ThreadedResolver because
aiodns did not support getaddrinfo until aio-libs/aiodns#118
see #559
bdraco added a commit to bdraco/aiodns that referenced this pull request Mar 30, 2024
This is a followup to aio-libs#118 to add `getnameinfo` as well to support

aio-libs/aiohttp#8270
aiohttp/resolver.py Outdated Show resolved Hide resolved
bdraco added a commit to bdraco/aiodns that referenced this pull request Mar 30, 2024
This is a followup to aio-libs#118 to add `getnameinfo` as well to support

aio-libs/aiohttp#8270
aiohttp/resolver.py Outdated Show resolved Hide resolved
@bdraco
Copy link
Member Author

bdraco commented Mar 30, 2024

I've tested this on production and everything works as expected. I'll do another turn and address review comments after the aiodns release

saghul pushed a commit to aio-libs/aiodns that referenced this pull request Mar 30, 2024
This is a followup to #118 to add `getnameinfo` as well to support

aio-libs/aiohttp#8270
@bdraco
Copy link
Member Author

bdraco commented Mar 30, 2024

Functional testing looks good, but all the tests for the resolver are patching the old code path so lots to update

aiohttp/resolver.py Outdated Show resolved Hide resolved
aiohttp/abc.py Fixed Show fixed Hide fixed
Copy link

codecov bot commented Mar 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.58%. Comparing base (742a7d9) to head (c3e3f40).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8270      +/-   ##
==========================================
+ Coverage   97.56%   97.58%   +0.01%     
==========================================
  Files         107      107              
  Lines       32987    33062      +75     
  Branches     3853     3865      +12     
==========================================
+ Hits        32184    32262      +78     
+ Misses        588      586       -2     
+ Partials      215      214       -1     
Flag Coverage Δ
CI-GHA 97.49% <100.00%> (+0.01%) ⬆️
OS-Linux 97.15% <99.14%> (+0.01%) ⬆️
OS-Windows 95.65% <67.52%> (-0.06%) ⬇️
OS-macOS 96.86% <99.14%> (+0.01%) ⬆️
Py-3.10.11 95.50% <67.52%> (-0.06%) ⬇️
Py-3.10.13 96.71% <99.14%> (+0.01%) ⬆️
Py-3.10.14 96.93% <99.14%> (+0.01%) ⬆️
Py-3.11.8 97.15% <100.00%> (+0.01%) ⬆️
Py-3.12.2 97.28% <100.00%> (+0.01%) ⬆️
Py-3.8.10 95.43% <54.70%> (-0.11%) ⬇️
Py-3.8.18 96.81% <74.35%> (-0.08%) ⬇️
Py-3.9.13 95.47% <67.52%> (-0.06%) ⬇️
Py-3.9.18 96.68% <99.14%> (+0.01%) ⬆️
Py-3.9.19 96.90% <99.14%> (+0.01%) ⬆️
Py-pypy7.3.15 96.47% <99.14%> (+0.01%) ⬆️
VM-macos 96.86% <99.14%> (+0.01%) ⬆️
VM-ubuntu 97.15% <99.14%> (+0.01%) ⬆️
VM-windows 95.65% <67.52%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bdraco added a commit to home-assistant/core that referenced this pull request Mar 31, 2024
This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270
bdraco added a commit to home-assistant/core that referenced this pull request Mar 31, 2024
This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270
@bdraco bdraco marked this pull request as ready for review March 31, 2024 19:24
@bdraco bdraco requested a review from asvetlov as a code owner March 31, 2024 19:24
@Dreamsorcerer
Copy link
Member

I would like to backport this fix to 3.9

Are we going to break anything if someone is using an older aiodns version? I'd prefer to avoid changing the dependency requirements in a patch version.

Technically, if we don't change our API, it's not a reason for us to include this in a minor or major version bump. SemVer-wise, that is. Fulfilling the dependencies is a job of a dependency resolver/installer. Similar to setting a specific python_requires value that wouldn't be an API change and wouldn't be breaking from this perspective.

Sure, I just think it's preferable to minimise requirement changes in order to get security/bug fixes. I'll let you two decide, I don't mind too much.

@bdraco
Copy link
Member Author

bdraco commented Mar 31, 2024

The interface is flexible enough that anyone who needs to fix can copy the code until it's released so I'm fine with leaving it in the current state for 3.9. At least that's the plan for Home Assistant, and we will delete the temporary backport after 3.10

It looks like anyone using it already has a workaround in place of setting AF_INET instead of unspecified so they have been living with this issue for years anyways. That's not great for IPv6 adoption but they also have ThreadedResolver even if its performance is worse so there are viable workarounds here now.

docs/conf.py Outdated Show resolved Hide resolved
bdraco and others added 3 commits March 31, 2024 13:52
Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <sviat@redhat.com>
@bdraco
Copy link
Member Author

bdraco commented Apr 3, 2024

I've been running this production for a few days now and everything is working well.

Unless there are any additional review comments, or reason we need to wait, I'll merge this later today.

@bdraco
Copy link
Member Author

bdraco commented Apr 4, 2024

Got distracted yesterday with the HA release so I didn't have time to get back to this.

I'm going to push this to another production system to be sure everything is OK shortly before merging

@bdraco
Copy link
Member Author

bdraco commented Apr 5, 2024

So far so good on both production systems.

Will check it again after dinner before merging.

@bdraco
Copy link
Member Author

bdraco commented Apr 5, 2024

All good

Also good with IPv6 disabled in the kernel.

I can't think of any other way to beat this up. So I'm happy with it now

@bdraco bdraco merged commit 012f986 into master Apr 5, 2024
36 of 37 checks passed
@bdraco bdraco deleted the aiodns_fixes branch April 5, 2024 03:28
Copy link
Contributor

patchback bot commented Apr 5, 2024

Backport to 3.10: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 012f986 on top of patchback/backports/3.10/012f9862d3fcb06d0b6c1abe394841c12ffe2b52/pr-8270

Backporting merged PR #8270 into master

  1. Ensure you have a local repo clone of your fork. Unless you cloned it
    from the upstream, this would be your origin remote.
  2. Make sure you have an upstream repo added as a remote too. In these
    instructions you'll refer to it by the name upstream. If you don't
    have it, here's how you can add it:
    $ git remote add upstream https://github.com/aio-libs/aiohttp.git
  3. Ensure you have the latest copy of upstream and prepare a branch
    that will hold the backported code:
    $ git fetch upstream
    $ git checkout -b patchback/backports/3.10/012f9862d3fcb06d0b6c1abe394841c12ffe2b52/pr-8270 upstream/3.10
  4. Now, cherry-pick PR Fix AsyncResolver to match ThreadedResolver behavior #8270 contents into that branch:
    $ git cherry-pick -x 012f9862d3fcb06d0b6c1abe394841c12ffe2b52
    If it'll yell at you with something like fatal: Commit 012f9862d3fcb06d0b6c1abe394841c12ffe2b52 is a merge but no -m option was given., add -m 1 as follows instead:
    $ git cherry-pick -m1 -x 012f9862d3fcb06d0b6c1abe394841c12ffe2b52
  5. At this point, you'll probably encounter some merge conflicts. You must
    resolve them in to preserve the patch from PR Fix AsyncResolver to match ThreadedResolver behavior #8270 as close to the
    original as possible.
  6. Push this branch to your fork on GitHub:
    $ git push origin patchback/backports/3.10/012f9862d3fcb06d0b6c1abe394841c12ffe2b52/pr-8270
  7. Create a PR, ensure that the CI is green. If it's not — update it so that
    the tests and any other checks pass. This is it!
    Now relax and wait for the maintainers to process your pull request
    when they have some cycles to do reviews. Don't worry — they'll tell you if
    any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

bdraco added a commit that referenced this pull request Apr 5, 2024
Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко) <sviat@redhat.com>
(cherry picked from commit 012f986)
bdraco added a commit that referenced this pull request Apr 5, 2024
Co-authored-by: Sviatoslav Sydorenko (Святослав Сидоренко)
balloob pushed a commit to home-assistant/core that referenced this pull request Apr 6, 2024
* Bump aiodns to 3.2.0

changelog: aio-libs/aiodns@v3.1.1...v3.2.0

* Switch to using the AsyncResolver with aiohttp

This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270

* Switch to using the AsyncResolver with aiohttp

This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270

* Switch to using the AsyncResolver with aiohttp

This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270

* Switch to using the AsyncResolver with aiohttp

This avoids creating executor jobs to do DNS resolution

AsyncResolver was not usable before aio-libs/aiohttp#8270
because it did not fallback to fallback to A records and only returned AAAA records
in most cases when IPv6 was available

This is a backport of aio-libs/aiohttp#8270

* fixes

* fix mocking in next_dns

* fix unmocked calls in blink

* more mocking fixes

* more fixes

* more fixes

* Fix missing mocking in nextdns tests

extracted from #114539

* extract from context
@bdraco bdraco mentioned this pull request Jul 22, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-3.10 Trigger automatic backporting to the 3.10 release branch by Patchback robot bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants