test_sequencer_basic failure

[dmacks]

Describe the bug
test_sequencer_basic fails with:

Traceback (most recent call last):
  File "/sw/build.build/distlib-py310-0.3.4-1/distlib-0.3.4/tests/test_util.py", line 410, in test_sequencer_basic
    self.assertIn(actual, expected)
AssertionError: [('upload_sdist',), ('sdist',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('register',), ('check',)] not found in ([('test',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], [('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('upload_bdist',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], [('upload_sdist',), ('sdist',), ('install',), ('install_distinfo',), ('install_data',), ('upload_bdist',), ('register',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('test',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)])

That's with python-3.10.2, and the same thing happens with python-3.9.10 and python-3.8.12. It passes on python 2.7/3.6/3.7.

To Reproduce
Steps to reproduce the behavior:

1. python3.10 setup.py build
2. export SKIP_ONLINE=1
3. python3.10 setup.py test
4. See error

Expected behavior
All tests succeed (pass or skip)

Environment

• OS X 10.13, python interpretters supplied via fink
• distlib-0.3.4 (also tried 0.3.3)

Additional information
There is a similar bug report at
https://bugzilla.redhat.com/show_bug.cgi?id=2033200

[dmacks]

This is weird...it succeeds sometimes (seems RedHat has the same situation?). Without installing or removing any other pythonmods. Maybe there's a timing/race-condition, or relying on some non-deterministic ordering?

[pradyunsg]

[('upload_sdist',), ('sdist',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('register',), ('check',)]
not found in
(
  [('test',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], 
  [('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('upload_bdist',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], 
  [('upload_sdist',), ('sdist',), ('install',), ('install_distinfo',), ('install_data',), ('upload_bdist',), ('register',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], 
  [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('test',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], 
  [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)]
)

Looks like it is sensitive to the ordering of the input.

See

distlib/tests/test_util.py

Lines 285 to 286 in d0e3f49

	# Note: these tests are sensitive to dictionary ordering
	# but work under Python 2.6, 2.7, 3.2, 3.3, 3.4 and PyPy 2.5

[stratakis]

So what would be the ideal solution here?

[hroncok]

Would something like this work?

self.assertIn(set(actual), (set(e) for e in expcted))

Or are there duplicate entries?

self.assertIn(sorted(actual), (sorted(e) for e in expcted))

[hroncok]

>>> actual = [('upload_sdist',), ('sdist',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('register',), ('check',)]
>>> expected = (
...   [('test',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], 
...   [('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('upload_bdist',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], 
...   [('upload_sdist',), ('sdist',), ('install',), ('install_distinfo',), ('install_data',), ('upload_bdist',), ('register',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], 
...   [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('test',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], 
...   [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)]
... )
>>> actual in expected
False
>>> set(actual) in [set(e) for e in expected]
True
>>> sorted(actual) in [sorted(e) for e in expected]
True

[hroncok]

Trying a patch.

[vsajip]

Run your tests with PYTHONHASHSEED=0 in the environment. Note how the GitHub actions step "Test with unittest" is configured here.

[vsajip]

Or run your tests with tox - tox.ini is appropriately configured.

[dmacks]

PYTHONHASHSEED=0 fixed it. Thanks! I prefer that to tox because tox itself has a bunch of python library dependencies, and I prefer to avoid putting packages like distlib higher in the dep-tree than necessary.

[hroncok]

PYTHONHASHSEED=0 does not fix this on Python 3.11.

$ tox -e py311
GLOB sdist-make: .../pypa/distlib/setup.py
py311 recreate: .../pypa/distlib/.tox/py311
py311 inst: .../pypa/distlib/.tox/.tmp/package/1/distlib-0.3.5.dev0.zip
py311 installed: distlib @ file://.../pypa/distlib/.tox/.tmp/package/1/distlib-0.3.5.dev0.zip
py311 run-test-pre: PYTHONHASHSEED='0'
py311 run-test: commands[0] | python setup.py test
.../pypa/distlib/setup.py:8: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.core
running test
............................s......................................................s.......s..........................sss.ss..........sss......ss.........................s..........s..F......................ssss.........ss.............
======================================================================
FAIL: test_sequencer_basic (test_util.UtilTestCase.test_sequencer_basic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../pypa/distlib/tests/test_util.py", line 410, in test_sequencer_basic
    self.assertIn(actual, expected)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: [('upload_sdist',), ('sdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('upload_bdist',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('register',), ('check',)] not found in ([('test',), ('upload_bdist',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], [('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('upload_bdist',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('upload_sdist',), ('sdist',), ('register',), ('check',)], [('upload_sdist',), ('sdist',), ('install',), ('install_distinfo',), ('install_data',), ('upload_bdist',), ('register',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('test',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)], [('upload_sdist',), ('sdist',), ('upload_bdist',), ('register',), ('install',), ('install_distinfo',), ('install_data',), ('install_scripts',), ('install_lib',), ('install_headers',), ('test',), ('build',), ('build_scripts',), ('build_py',), ('build_ext',), ('build_clibs',), ('check',)])

----------------------------------------------------------------------
Ran 235 tests in 19.275s

FAILED (failures=1, skipped=21)
ERROR: InvocationError for command .../pypa/distlib/.tox/py311/bin/python setup.py test (exited with code 1)
____________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________
ERROR:   py311: commands failed

[dmacks]

Oops; I'm only on py310:(

[vsajip]

Should be fixed in 3be4cc0.