Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Functions executed via multiprocessing aren't reliably being covered by coverage.py #1825

Open
cacti77 opened this issue Jul 29, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@cacti77
Copy link

cacti77 commented Jul 29, 2024

Describe the bug
We have code that is run in Azure DevOps using Python's multiprocessing library executed via Python's built-in unittest module. With print() statements in the parallel code we can show that the code is consistently executed. However, coverage.py reports that these functions are covered in some pipeline runs but not others, even though no code change has happened between the runs. This means code coverage reports aren't reliable and could potentially affect our quality gate criteria for passing builds.

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

What version of Python are you using?
Python 3.8.10 on Microsoft Windows Server 2022 (Azure DevOps runner image windows-2022, with this software).

What version of coverage.py shows the problem? The output of coverage debug sys is helpful.
Coverage.py, version 7.6.0 with C extension

What versions of what packages do you have installed? The output of pip freeze is helpful.

Package             Version
------------------- -----------
bokeh               3.1.1
contourpy           1.1.1
coverage            7.6.0
cycler              0.12.1
et-xmlfile          1.1.0
fonttools           4.53.1
greenlet            3.0.3
importlib_resources 6.4.0
Jinja2              3.1.4
joblib              1.4.2
kiwisolver          1.4.5
lxml                5.2.2
MarkupSafe          2.1.5
matplotlib          3.7.5
numpy               1.24.4
openpyxl            3.1.5
packaging           24.1
pandas              2.0.3
pillow              10.4.0
pip                 24.1.2
pyparsing           3.1.2
python-dateutil     2.9.0.post0
pytz                2024.1
PyYAML              6.0.1
scikit-learn        1.3.2
scipy               1.10.1
setuptools          71.1.0
six                 1.16.0
SQLAlchemy          2.0.31
threadpoolctl       3.5.0
tomli               2.0.1
tomli_w             1.0.0
tornado             6.4.1
typing_extensions   4.12.2
tzdata              2024.1
wheel               0.43.0
xyzservices         2024.6.0
zipp                3.19.2

Plus our own package we've installed (redacted).

What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
We can't share the code unfortunately, but it was launched using Python's multiprocessing module as per here. The DevOps Windows VMs have 2 CPUs.

What commands should we run to reproduce the problem? Be specific. Include everything, even git clone, pip install, and so on. Explain like we're five!
Print() statements in the function called with mp.Pool.map() show that that the function is being called consistently in parallel, but the HTML coverage report only shows that the function has coverage in some DevOps runs, but not others.

Expected behavior
As the parallel code is always being executed the function should always show coverage.

Additional context
Add any other context about the problem here.
In the DevOps pipeline we execute numerous test modules in separate folders like this:
coverage run --rcfile=.coveragerc -m unittest --failfast --verbose test_module1.py

The .coveragerc config file looks like this (with package name and omit folders obfuscated):

[run]
source_pkgs =
    our_package

omit =
    */a/b/*

branch = true

concurrency =
    multiprocessing

parallel = true

After all the tests we run coverage combine, passing it all the folder names where the tests were executed. Then run coverage report and coverage xml. We also use sed to edit coverage.xml to make the file paths more human readable. Finally we publish the edited xml file to the pipeline's artifacts using DevOps' PublishCodeCoverageResults@2 task. Code Coverage then appears in the build's results and the HTML report can be drilled down into on a per file basis.

The problem is that when we drill down into the Python module that is executed in parallel, sometimes it is shown as having been covered (green) but other times it is not (red). This is despite print() statements in the parallel code showing that the code has been executed even when coverage.py thinks it has not.

@cacti77 cacti77 added the bug Something isn't working label Jul 29, 2024
@cacti77 cacti77 changed the title Functions executed via multiprocessing aren't consistently being covered by coverage.py Functions executed via multiprocessing aren't reliably being covered by coverage.py Jul 29, 2024
@cacti77
Copy link
Author

cacti77 commented Jul 29, 2024

I notice a similar issue about undeterministic results at #1533, albeit using pytest rather than unittest. @nedbat I apologise this issue doesn't have a minimal reproducible for you, but I thought you should know about this anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant