[BUG] Parsing of __init__.py fails to reach _read_utf8_with_fallback at the end of file #4399
Comments
autogris
changed the title
|
Also getting on M3 Pro |
|
I believe that what is happening is a circular reference: the library calls pkg_resources to load an entry-point thar itself calls pkg_resources... (this is a hypothesis, I haven't checked). We can avoid that by changing pkg_resources to define that method earlier in the file... However, the difficulty part here is to create a regression test based on a minimal reproducible example that does not depend on an external package. Can anyone help to create such regression test to be added to the test suite to avoid similar problems in the future? |
abravalheri
added
help wanted
Needs Simplified Reproducer
|
@autogris, @areebsiddiquidbz, how did you install deluge? If that is the case, what happens if you uninstall that package and install it with the recommended method ( |
|
I tried a couple of tests with diff --git i/pkg_resources/tests/test_working_set.py w/pkg_resources/tests/test_working_set.py
index 57f62b549..bc685bf0e 100644
--- i/pkg_resources/tests/test_working_set.py
+++ w/pkg_resources/tests/test_working_set.py
@@ -1,9 +1,14 @@
+import functools
import inspect
+import os
import re
+import shutil
+import subprocess
import textwrap
-import functools
+import venv
import pytest
+import jaraco.path
import pkg_resources
@@ -499,3 +504,95 @@ def test_working_set_resolve(
resolve_call()
else:
assert sorted(resolve_call()) == sorted(resolved_dists_or_exception)
+
+
+class TestInteropDeprecatedEgglink:
+ """The issue pypa/setuptools#4399 reports a problem when running
+ ``import pkg_resources`` and running ``WorkingSet._build_master`` in ``_call_aside``
+ and ``resolve_egg_link``.
+ """
+
+ @pytest.fixture
+ def venv_python(self, tmp_path):
+ venv.create(tmp_path / ".venv", clear=True, with_pip=False)
+ search_path = os.pathsep.join(map(str, tmp_path.glob(".venv/*/")))
+ return shutil.which("python", path=search_path)
+
+ @pytest.fixture
+ def site_packages(self, venv_python, tmp_path):
+ getsitepackages = (
+ "import os, site;"
+ "print(next(p for p in site.getsitepackages() if os.path.isdir(p)))"
+ )
+ return subprocess.check_output(
+ [venv_python, "-c", getsitepackages],
+ text=True,
+ encoding="utf-8",
+ cwd=tmp_path,
+ ).strip()
+
+ @pytest.fixture
+ def install_pkg_resources(self, venv_python, site_packages, tmp_path, request):
+ """Install this version of pkg_resources and setuptools"""
+ with open(os.path.join(site_packages, "this.pth"), "w", encoding="utf-8") as f:
+ f.write(f"{request.config.rootdir}\n") # Optimistic UTF-8 encoding
+
+ subprocess.run( # sanity check
+ [venv_python, "-Wignore", "-c", "import pkg_resources, setuptools"],
+ cwd=tmp_path,
+ check=True,
+ )
+
+ @pytest.fixture
+ def simple_pkg(self, tmp_path):
+ files = {
+ "pkg_root": {
+ "pkg": {
+ "__init__.py": "def hello(): print('world')",
+ "other.py": "import pkg_resources\ndef world(): print('hello')",
+ },
+ "setup.cfg": inspect.cleandoc(
+ """
+ [options.entry_points]
+ console_scripts =
+ pkg = pkg.other:world
+ pkg.other =
+ pkg = pkg.other:world
+ """
+ ),
+ },
+ }
+ jaraco.path.build(files, prefix=tmp_path)
+ return tmp_path / "pkg_root"
+
+ @pytest.fixture
+ def egg_link(self, simple_pkg, venv_python, install_pkg_resources, tmp_path):
+ """Install ``simple_pkg`` using an egg-link, via setuptools' develop"""
+ _ = install_pkg_resources # install setuptools
+ cmd = "__import__('setuptools').setup()"
+ subprocess.run(
+ [venv_python, "-Wignore", "-c", cmd, "develop"], cwd=simple_pkg, check=True
+ )
+ return next(tmp_path.rglob("*.egg-link"))
+
+ def test_global_working_set(
+ self, install_pkg_resources, egg_link, venv_python, tmp_path
+ ):
+ _ = install_pkg_resources
+ assert "/.venv/" in str(egg_link).replace(os.sep, "/")
+ assert egg_link.name == "pkg.egg-link"
+ assert egg_link.exists()
+
+ cmd = "print('hello'); import pkg; pkg.hello()"
+ out = subprocess.check_output(
+ [venv_python, "-Wignore", "-c", cmd],
+ text=True,
+ encoding="utf-8",
+ cwd=tmp_path,
+ )
+ print(out)
+ assert out.strip() == "hello\nworld"
+
+ exe = shutil.which("pkg", path=os.path.dirname(venv_python))
+ out = subprocess.check_output(exe, text=True, encoding="utf-8", cwd=tmp_path)
+ assert out.strip() == "hello"
========================================= test session starts ==========================================
platform linux -- Python 3.8.10, pytest-8.2.1, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /home/abravalheri/workspace/setuptools
configfile: pytest.ini
plugins: checkdocs-2.13.0, enabler-3.1.1, home-0.5.1, mypy-0.10.3, ruff-0.3.2, subprocess-1.5.0, timeout-2.3.1, xdist-3.6.1, typeguard-4.2.1
16 workers [41 items]
......................................... [100%]
================================================= mypy =================================================
Success: no issues found in 1 source file
========================================= 41 passed in 15.67s ==========================================
py: OK (26.80=setup[8.93]+cmd[17.87] seconds)
congratulations :) (27.49 seconds)
I still need help to create a minimal reproducer for this, so we can have a test for this in the test suite. |
|
I don't think I ever installed anything with The script for Deluge produces a package using this installation method, which then is searched for binaries and stripped with |
|
Ok, something I noticed in the traceback is that the last called .py from Deluge side is pluginmanagerbase.py. I do have plugins installed in the .egg format, which I guess are loaded before the window is draw. I will try later to uninstall all third party plugins, and report what happens. @areebsiddiquidbz, are you using third party plugins too? Quickly checking here, I can see in the source code that the oldest plugin hasn't been updated for many years and it contains this absolute_import line, which seems suspect (https://github.com/stefantalpalaru/deluge-default-trackers/blob/master/setup.py): The only other third party plugin is this https://github.com/ratanakvlun/deluge-ltconfig/blob/master/setup.py |
|
I've removed the plugins and also tried to recreate the eggs with |
I also tried to create a minimal reproducer using This means that the minimal reproducible example that I am trying to write still needs more details, and I am not managing to get it right. Specifically, I did not manage to find the condition that triggers the circular references... I would appreciate any help writing such minimal reproducer. (my work on the reproducer so far)diff --git c/setuptools/tests/environment.py w/setuptools/tests/environment.py
index b9de4fda6..cb1f850d8 100644
--- c/setuptools/tests/environment.py
+++ w/setuptools/tests/environment.py
@@ -1,3 +1,4 @@
+import functools
import os
import sys
import subprocess
@@ -5,6 +6,7 @@ import unicodedata
from subprocess import Popen as _Popen, PIPE as _PIPE
import jaraco.envs
+from path import Path
class VirtualEnv(jaraco.envs.VirtualEnv):
@@ -33,6 +35,16 @@ class VirtualEnv(jaraco.envs.VirtualEnv):
kwargs["env"] = env
return subprocess.check_output(cmd, *args, **kwargs)
+ @functools.cached_property
+ def site_packages(self):
+ prefix = os.path.abspath(self.dir)
+ getsitepackages = (
+ "import os, site;"
+ "opts = site.getsitepackages();"
+ f"print(next(p for p in opts if os.path.isdir(p) and {prefix!r} in p))"
+ )
+ return Path(self.run(["python", "-I", "-c", getsitepackages]).strip())
+
def _which_dirs(cmd):
result = set()
diff --git c/setuptools/tests/test_bdist_egg.py w/setuptools/tests/test_bdist_egg.py
index 12ed4d328..cf8499436 100644
--- c/setuptools/tests/test_bdist_egg.py
+++ w/setuptools/tests/test_bdist_egg.py
@@ -1,10 +1,12 @@
"""develop tests"""
+import inspect
import os
import re
import zipfile
import pytest
+import jaraco.path
from setuptools.dist import Distribution
@@ -67,3 +69,77 @@ class Test:
names = list(zi.filename for zi in zip.filelist)
assert 'hi.pyc' in names
assert 'hi.py' not in names
+
+
+class TestInteropPkgResources:
+ """The issue pypa/setuptools#4399 reports a problem when running
+ ``import pkg_resources`` and running ``WorkingSet._build_master`` in ``_call_aside``
+ and ``resolve_egg_link``.
+ """
+
+ @pytest.fixture
+ def simple_pkg(self, tmp_path):
+ files = {
+ "pkg_root": {
+ "pkg": {
+ "__init__.py": "def hello(): print('world')",
+ "other.py": "import pkg_resources\ndef world(): print('hello')",
+ },
+ "setup.cfg": inspect.cleandoc(
+ """
+ [options.entry_points]
+ console_scripts =
+ pkg = pkg.other:world
+ pkg.other =
+ pkg = pkg.other:world
+ """
+ ),
+ },
+ }
+ jaraco.path.build(files, prefix=tmp_path)
+ return tmp_path / "pkg_root"
+
+ @pytest.fixture
+ def egg(self, simple_pkg, venv):
+ """Install ``simple_pkg`` using an egg"""
+ cmd = ["python", "-Wignore", "-c", "__import__('setuptools').setup()"]
+ venv.run([*cmd, "bdist_egg"], cwd=simple_pkg)
+ file = next(simple_pkg.glob("dist/*.egg"))
+ venv.run([*cmd, "easy_install", "--prefix", venv.dir, file], cwd=simple_pkg)
+ return venv.site_packages.glob("*.egg")[0]
+
+ PY_C = ["python", "-I", "-Wignore", "-c"]
+
+ def test_installed_egg_sanity_check(self, egg, venv):
+ assert f"{venv.site_packages}/" in str(egg).replace(os.sep, "/")
+ assert egg.exists()
+ out = venv.run([*self.PY_C, "import pkg; print(pkg.__path__)"]).strip()
+ assert f"{venv.site_packages}/" in out
+ assert ".egg" in out
+
+ @pytest.mark.parametrize(
+ "cmd, output",
+ [
+ ([*PY_C, "import pkg; pkg.hello()"], "world"),
+ ([*PY_C, "import pkg_resources;from pkg.other import *;world()"], "hello"),
+ (["pkg"], "hello"),
+ (
+ [
+ *PY_C,
+ inspect.cleandoc(
+ """
+ import pkg_resources
+ ep = next(pkg_resources.iter_entry_points("pkg.other", "pkg"))
+ fn = ep.load()
+ fn()
+ """
+ ),
+ ],
+ "hello",
+ ),
+ ],
+ )
+ def test_global_working_set(self, egg, venv, cmd, output):
+ out = venv.run(cmd).strip()
+ print(out)
+ assert out == outputpipx run --python python3.11 tox -- -p no:cov -p no:perf -p no:mypy -x -n 4 setuptools/tests/test_bdist_egg.py
# ...
16 workers [9 items]
......... [100%]
========================================== 9 passed in 24.95s ==========================================
py: OK (30.36=setup[3.15]+cmd[27.21] seconds)
congratulations :) (31.68 seconds) |
|
Sorry, I'm not really familiar with python. I could try to inspect Deluge code and see if I can identify where the thing is triggered, but I'm not sure, it seems fairly complex to my limited knowledge. I haven't seen this problem in any other python application, so maybe it is really specific to how Deluge call this library. It does not seem to be related to the .egg plugins since I've tried to run with all plugins removed and still got the same error. |
|
Hi. I am seeing this same |
|
Hi @mcayanan , please see the discussion above that describes what is missing for us to act on this particular problem. A way to expedite the fix would be to provide a small minimal reproducer example that do not rely on complex and existing packages so that it can help us to understand the real root cause of the problem. Are you interested in contributing with one? |
|
Hi @abravalheri . I myself am still trying to understand root cause. I will be out of the office starting on 6/6 and will return on 6/17, so I can revisit this at that time to investigate more and provide feedback if I'm able to easily reproduce it. |
|
Ok, I've done some digging and came up with a minimal reproducer: https://github.com/NiklasMM/setuptools-minimal-reproducer It seems to have something to do with the dependency zope-interface, which we have through twisted and Deluge, which was in the original report, also seems to have. I'm not quite sure why, but it can be defined as I could only observe this behavior in Python 3.10, not it 3.12. (Others I haven't tested) That's how war I could narrow it down so far. Hope it helps. |
|
Hi @NiklasMM, thank you very much for having a look on this. When I run the following: > docker run --rm -it python:3.11-bookworm /bin/bash
git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
python -m venv /tmp/.venv
/tmp/.venv/bin/python -m pip install -e .
/tmp/.venv/bin/mypackageI still get "Print me if you can". No errors raised. |
|
Could you try again with python:3.10-bookworm? It might be limited to 3.10 Also make sure to explicitly update setuptools to 70.0 |
|
Oh I see, no Thank you! |
|
Great idea to reproduce it in docker! I can confirm the following works as a reproduction path in 3.10, 3.11 and 3.12 |
|
Thank you, I can also see that this seems to be an "editable" install problem only: git clone https://github.com/NiklasMM/setuptools-minimal-reproducer.git /tmp/repro
cd /tmp/repro
pip install -U setuptools
pip install .
mypackage
# => Print me if you can |
|
If we try to make OR OR So I suppose this only happens when |
|
Thank you very much @NiklasMM. I still haven't managed to get into the bottom of the problem (e.g. there are some open questions: "why But it is enough to write a test case that verifies that the fix work. So I have created #4422. On a side note, this seems related to a combination of 2 deprecated things: deprecated legacy-style namespace packages and deprecated non-PEP 517 builds, so that is probably why the problem was not perceived before. Also, in the future the implementation may change as setuptools only guarantees limited best-effort backwards compatibility for deprecated features. |
|
Version 70.1.0 should not cause this exception |
When running Deluge 2.1.1.dev99 (deluge-torrent/deluge@7f3f7f6) with Setuptools 70.0.0, an error occurs claiming that _read_utf8_with_fallback is not defined in setuptools' init.py. By moving the function position to the head of the referred file, after the imports, the program runs normally. This error does not occur on Setuptools 69.x and earlier.
System: linux x64, kernel 6.9.0
Python: 3.11.9
Discussed in #4398
Originally posted by autogris May 24, 2024
Hi. I'm trying to debug an error that emerged since upgrading setuptools to 70.0.0. When running Deluge, it gives me the following error:
I've tried to recompile Deluge, reinstall all python modules, to no success. The supposedly undefined function is in fact present at
python3.11/site-packages/setuptools/unicode_utils.py. Is this a bug on setuptools side or Deluge? I'm on Python 3.11.9The text was updated successfully, but these errors were encountered: