Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
for compatibility with Python 3.10.
- REL: add `pytest >= 4.6.11` to the argument
`tests_require` of the function `setuptools.setup`
in the script `setup.py`.
This lower bound has been selected to
ensure compatibility with both Python 3 and
Python 2.7. See below for details.
- REL: add `pytest >= 4.6.11` to file `requirements.txt`
- REL: remove `nose` from argument `tests_require` of
the function `setuptools.setup` in script `setup.py`
- REL: remove `nose` from file `requirements.txt`
- CI: update file `.travis.yml`
- CI: remove the collection of
coverage measurements on Travis CI,
because these measurements do not collect
Cython coverage with the build configuration
currently used for CI testing.
- DEV: remove `coveralls` from the
file `requirements.txt`, because the
package `coveralls` was used only on Travis CI.
- TST: add a configuration file `tests/pytest.ini`
and include it in `MANIFEST.in`
- GIT: ignore `.pytest_cache/` in `.gitignore`
## Motivation
The change from [`nose == 1.3.7`](
https://pypi.org/project/nose/1.3.7/#history)
to [`pytest`](
https://pypi.org/project/pytest)
is motivated by compatibility with Python 3.10.
## `nose` is incompatible with Python 3.10
The package `nose`, which was used to run
the tests of `dd`, is not compatible with Python 3.10
(for details, read the last section below).
Also, `nose` uses the `imp` module from
CPython's standard library.
The module `imp` [is deprecated](
https://docs.python.org/3.10/library/imp.html),
so it may be removed in some future Python version.
## Summary of transition to `pytest`
In summary, using `pytest` with the
existing tests requires adding
a [configuration file `tests/pytest.ini`](
https://docs.pytest.org/en/latest/reference/customize.html#configuration-file-formats)
to tell `pytest` which functions, classes, and
methods to collect tests from
(called "discovery" of tests).
The [parameter `--continue-on-collection-errors`](
https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags)
tells `pytest` to not stop in case any
test module fails to import,
and to continue with running the tests.
The ability to run the tests when some
`dd` C extension modules are not installed is necessary.
After transitioning the tests to `pytest`,
the tests have been confirmed
to run successfully:
- on Python 2.7 with `pytest == 4.6.11`, and
- on Python 3.9 with `pytest == 6.2.4`.
## Considering using `unittest`
For writing `dd` tests, the module [`unittest`](
https://docs.python.org/3/library/unittest.html)
(part of CPython's standard library) suffices.
For *discovering* the tests, `unittest` seems to
require that tests be methods of subclasses of
the class `unittest.TestCase`.
This is not the case in the tests of `dd` tests.
Using `pytest` allows changing the test runner from `nosetests`
with minimal changes to the tests themselves.
Test discovery using `unittest` could possibly be
implemented by adding a file `tests/__init__.py`, and
defining in that file a function `load_tests`,
following the [documentation of `unittest`](
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover).
In any case, it is simpler to use `pytest`,
which requires only a configuration file.
If `unittest` encounters an `ImportError` during
collection of the tests (i.e., when `unittest` tries to
import test modules), then it stops.
There does not appear to be any way to tell `unittest`
to continue and run the rest of the test modules
(those that *could* be imported).
## Usage of `nose`
The dependence on `nose` was minimal.
Only one function was used from `nose`:
the function `nose.tools.assert_raises`.
The function `assert_raises` is dynamically defined
[in the module `nose.tools.trivial`](
https://github.com/nose-devs/nose/blob/release_1.3.7/nose/tools/trivial.py#L32-L54)
by instantiating the class `unittest.TestCase`,
and setting `assert_raises` to equal the
[bound method `assertRaises`](
https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises)
of the instance of `TestCase`.
So the function `assert_raises` from `nose` is
a PEP8-compliant binding for the method
`unittest.TestCase.assertRaises`.
Reading the code of `unittest`:
- <https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L156-L243>
- <https://github.com/python/cpython/blob/6fdc4d37f3fdbc1bd51f841be6e5e4708a3b8798/Lib/unittest/case.py#L704-L735>
it follows that the usage:
```python
with nose.tools.assert_raises(AssertionError):
foo(1, 2)
```
is equivalent to the following code
(`AssertionError` here is used as an example):
```python
with unittest.TestCase().assertRaises(AssertionError):
foo(1, 2)
```
## Replacing usage of `nose` with `pytest` in test code
The [context manager `pytest.raises`](
https://docs.pytest.org/en/latest/reference/reference.html#pytest-raises)
is a [drop-in replacement](
https://en.wikipedia.org/wiki/Drop-in_replacement)
for the function `nose.tools.assert_raises`:
```python
with pytest.raises(AssertionError):
foo(1, 2)
```
Also, the tests can still be run with `nosetests`
on Python versions where `nose` is still available.
## Replacing the test runner `nosetests` with `pytest`
- `pytest` correctly recognized the
test files by default
- `pytest` does not recognize by default methods of
classes that do not start with "Test" as test methods.
The configuration file is necessary for
changing this behavior of `pytest`
(in particular, the command-line parameter `-k`
did not seem to work for classes).
Relevant documentation:
- <https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery>
- <https://docs.pytest.org/en/latest/example/pythoncollection.html#changing-naming-conventions>
- <https://docs.pytest.org/en/latest/how-to/nose.html>
- <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_classes>
- <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_functions>
- <https://docs.pytest.org/en/latest/reference/reference.html#confval-python_files>
- The call `pytest tests/foo_test.py` imports
the package `dd` from `site-packages`
(assuming that the module `foo_test.py` contains
the statement `import dd`).
So the default behavior of `pytest` is
as intended.
In contrast, `nosetests tests/foo_test.py` imports
the package `dd` from the local directory `dd/`,
even though `dd` *is* installed under `site-packages`.
In any case, `pytest` is called from within
the directory `tests/`, as was done for `nosetests`.
Both:
- `python -m pytest tests/foo_test.py` and
- `PYTHONPATH=. pytest tests/foo_test.py`
result in importing `dd` from the local directory `dd/`.
Relevant documentation:
- <https://docs.pytest.org/en/latest/explanation/pythonpath.html#invoking-pytest-versus-python-m-pytest>
- <https://docs.pytest.org/en/latest/how-to/usage.html#invoke-python>
As remarked above, the parameter
[`--continue-on-collection-errors`](
https://docs.pytest.org/en/latest/reference/reference.html#command-line-flags)
of `pytest` needs to be used for running the tests
when some of the C extension modules are not installed.
For example:
```
cd tests/
pytest -v --continue-on-collection-errors .
```
To activate [Python's development mode](
https://docs.python.org/3/library/devmode.html):
```
cd tests/
python -X dev -m pytest -vvv --continue-on-collection-errors .
```
## Further remarks
Observations about `pytest`:
- detects assertions that failed, and marks
their source lines
- avoids the deprecated `imp` module (of the
standard library) that `nose` uses
(and thus the associated `DeprecationWarning`)
<https://docs.python.org/3.10/library/imp.html>
- running the tests of `dd` with `pytest` revealed
several `DeprecationWarning`s that were previously
not shown by `nose` (these warnings were about
invalid escape sequences due to
backslashes appearing in non-raw strings).
[`pytest == 6.2.4`](https://pypi.org/project/pytest/6.2.4) is not
compatible with Python 2.7.
[`pytest == 4.6.11`](https://pypi.org/project/pytest/4.6.11/)
is the latest version of `pytest` that is compatible with Python 2.7
(released on June 5, 2020).
`pytest` specifies `python_requires` [PEP 345](
https://www.python.org/dev/peps/pep-0345/#requires-python),
[PEP 503](
https://www.python.org/dev/peps/pep-0503/):
- <https://github.com/pytest-dev/pytest/blob/4.6.11/setup.cfg#L48>
- <https://github.com/pytest-dev/pytest/blob/5.0.0/setup.cfg#L43>
So including `pytest>=4.6.11` in the file
`requirements.txt` suffices to install,
on each Python version, the latest version of
`pytest` that is compatible with that Python version.
This simplifies testing on CI, and packaging.
In other words, conditional installations in
the file `.travis.yml` are not needed for `pytest`,
neither conditional definition of `tests_require`
in the script `setup.py`.
This approach leaves implicit the upper bound on
`pytest` in `tests_require`.
This upper bound is specified explicitly by
`pytest` itself, depending on the Python version of
the interpreter.
It appears that `pip == 9.0.0` and
`setuptools == 24.2.0` are required,
to correctly implement `python_requires`:
- <https://pip.pypa.io/en/stable/news/#v9-0-0>
- <https://setuptools.readthedocs.io/en/latest/history.html#v24-2-1>
## How replacing usage of `nose` with `unittest` would have looked like
A way to replace `nose` could have been to
add a module `tests/utils.py` containing:
```python
"""Common functionality for tests."""
import unittest
_test_case = unittest.TestCase()
assert_raises = _test_case.assertRaises
```
which is close to what `nose` does.
The function `assert_raises` could then
be imported from the module `utils` in
test modules, and used.
Using `pytest` avoids the need for
this workaround.
## Details about the incompatibility of `nose` with Python 3.10
[`nose == 1.3.7`](
https://pypi.org/project/nose/1.3.7/#history)
imports in Python 3.10. Running `nosetests` fails,
due to imports from the module `collections` of
classes that have moved to the module `collections.abc`.
Relevant information about CPython changes
in `collections` (removal of ABCs):
- <python/cpython#23754>
- <https://bugs.python.org/issue37324>
- <https://docs.python.org/3.10/library/collections.abc.html#collections-abstract-base-classes>
- Deprecation since Python 3.3,
present until Python 3.9:
<https://docs.python.org/3.9/library/collections.html#module-collections>
- Removed in Python 3.10:
<https://docs.python.org/3.10/whatsnew/3.10.html#removed>
## About skipping tests
The decorator `unittest.skipIf` is recognized
by `pytest`, and skipped tests are correctly
recorded and reported. In any case, note also
the `pytest` test-skipping facilities:
- <https://docs.pytest.org/en/latest/how-to/skipping.html>
- <https://docs.pytest.org/en/latest/how-to/unittest.html>
- <https://docs.pytest.org/en/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option>
## About passing `-X dev` to `python` in the `Makefile`
The argument `dev` is available for the `python` option
[`-X` only on Python 3.7 and higher](
https://docs.python.org/3/library/devmode.html#devmode).
So the `Makefile` rules where `-X dev` appears are
not compatible with ealier Python versions supported by `dd`.
This is not an issue: the development environment is
intended to be Python 3.9 or higher, so there is no issue
with using `-X dev`.
## Avoiding interaction between tests via class attributes
Avoid class attributes in test classes.
Use [data attributes][1] instead.
Initialize the data attributes in setup methods
of the test classes, as is common practice.
This approach avoids interaction (via class
attributes) between test scripts that
import the same modules of common tests.
With `nose`, this kind of interaction
apparently did not occur, as observed by
test failures that were expected to happen.
However, `pytest` apparently runs tests in
a way that changes to imported modules
(e.g., class attributes)
persist between different test scripts.
This `pytest` behavior was observed by
the disappearance of test failures when
running with `pytest` (the test failures
were observable with `pytest` only when
telling `pytest` to run individual test
scripts, instead of collecting tests from
all test scripts.
The cause of the issue with `pytest` was
the modification of class attributes
(not [data attributes][1]) from the
importing module, of classes in the
imported module.
The modifications were done by setting the
class attribute `DD` that defines the BDD or
ZDD manager class. Both the scripts
`cudd_test.py` and `autoref_test.py` made
modifications. The end result was
`autoref` tests being run using the
class `dd.cudd.BDD`.
Using data attributes, instead of class
attributes, and subclassing, avoids this
kind of erroneous testing.
This approach is explicit [PEP 20][4].
[1]: https://docs.python.org/3/tutorial/classes.html#instance-objects
[2]: https://github.com/pytest-dev/pytest-xdist
[3]: https://github.com/pytest-dev/pytest-forked
[4]: https://www.python.org/dev/peps/pep-0020/- Loading branch information
