pytest-cov seems to override the use of [paths] in .coveragerc, probably because it passes source= to the coverage API
#130
Comments
|
I also had issues with combining files for the purpose of normalizing tox coverage paths. It would run the tests, but the output would always show the paths for the files in the tox virtualenv, not the source file paths I wanted to see. I wanted it to replace these commands that I've seen commonly, and used myself. I eventually figured out that this could be done by enabling parallel mode from the coverage configuration (which I like to put in [coverage:run]
source = mypackage
parallel = True
[coverage:paths]
source =
src/
.tox/*/lib/python*/site-packages/Unfortunately, it still didn't work. It appears that somehow the combine call doesn't work the way that I, and I expect the author that originally implemented the feature, expected it to. I will be submitting a pull request shortly with the code change that I have found fixes the problem. I'm not clear on whether I'm using the coverage API correctly, but hopefully I can get feedback on it in a code review on that pull request. |
|
As promised: #142 |
|
@ceridwen note that @ryanhiebert in my projects it just works, although I have a bit different path configuration: [paths]
source =
src/packagename
*/site-packages/packagenameWhat exactly is wrong with the combine call? |
|
The problem with the combine call as-is is that it doesn't combine runs from parallel mode. Are you using tox and it "just works"? Note that this isn't for xdist, I'm not using parallel mode for that, but just to normalize the paths. The differences between our files are a) I only cared to watch for files installed in tox virtualenvs, and b) I don't mention the packagename, mostly because what I'm currently testing is just a module, not a package, and I think that would look weird. |
|
Yes yes, not xdist. This is the full config https://github.com/ionelmc/python-nameless/blob/master/.coveragerc and output https://travis-ci.org/ionelmc/python-nameless/jobs/188214771#L223 (paths are What am I missing here? |
|
I'm not sure, that looks basically identical to what I'm trying to do. I'll look at it closer in a little bit. Thank you, @ionelmc . |
|
@ionelmc : OK, there's two things that you're missing. First, you're using Second, it's not combining the files the way it should be. Once the combining is done, all the files left in parallel mode, with filenames like It's possible that you'd like there to be the option to not combine those parallel files immediately, for instance if you're running multiple tox jobs that will need to be combined with something after all of those jobs. If that's what you prefer, I can add a new flag or something to make that work. Normalizing the paths in that case probably isn't going to work without some more work to allow us to combine the combined files, since the path normalization happens during combining. However in my case, and I suspect most cases, it's more important to display the coverage the way it's expected than to make files easily combinable, since having it combine between tox envs is going to mean some kind of orchestration outside of tox, and in any case between pytest runs. So that you can see the real code that I'm working with, here's some links. Note that I don't have Travis set up to run these tests yet. I'm still working on other things, I wasn't planning on getting Travis and everything set up quite yet. Also note that all the interesting stuff is in the I'll just give a link to the The problem versionhttps://github.com/ryanhiebert/lektor-git/blob/d079bf0/tox.ini output (truncated, from tox): ls -a .coverage*: The paths aren't normalized, but it is showing that it's at 100% coverage. The file remaining is the parallel version of the file, not the desired Like namelesshttps://github.com/ryanhiebert/lektor-git/blob/8b339c0/tox.ini diff: diff --git a/tox.ini b/tox.ini
index fb5451c..6a50f6a 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,7 +12,7 @@ commands =
[coverage:run]
branch = True
-source = lektor_git
+source = src
parallel = True
[coverage:paths]output: ls -a coverage*: This is showing the paths I want, but the coverage is wrong, saying 0% instead of 100%. The file remaining is the parallel version of the file, not the desired With the patch from #142https://github.com/ryanhiebert/lektor-git/blob/14a0f36/tox.ini diff: diff --git a/tox.ini b/tox.ini
index fb5451c..aaa6e5b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -6,7 +6,7 @@ whitelist_externals = git
deps =
pytest
pytest-mock
- pytest-cov
+ git+https://github.com/ryanhiebert/pytest-cov.git@062590574c231f68648cbbfc029669843b519de6#egg=pytest-cov
commands =
pytest --cov {posargs}output: ls -a .coverage*: The paths are normalized the way I want, and the coverage is correctly showing 100%. The file remaining is the desired Sorry this got so long-winded. I wanted to clarify the problem for you, so I haven't yet looked more into the test failures. |
|
Oh yeah, now there's easy steps to reproduce the problem: # Make sure tox is installed
git clone git@github.com:ryanhiebert/lektor-git.git
cd lektor-git
git checkout d079bf0 # git hash in case I push more commits
tox |
|
This does the trick: diff --git a/tox.ini b/tox.ini
index fb5451c..4f56c00 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,10 +12,10 @@ commands =
[coverage:run]
branch = True
-source = lektor_git
+source = src
parallel = True
[coverage:paths]
source =
- src/
- .tox/*/lib/python*/site-packages/
+ src/lektor_git.py
+ */site-packages/lektor_git.pyYou have a point about the stray |
|
With the patch you provided above, it is still showing incorrect coverage, because it's not measuring the files are are run, it's just measuring the files at the path. But the files at the path aren't run, because they have been installed into tox's virtualenv. Here's the report output. Notice that the coverage is 0%, not the correct 100%. The parallel coverage file is just a symptom of the issue, which is that files are not being combined. If they were being combined, then we'd have a proper BTW, if you just install to a virtualenv manually instead of using tox, you'll hit this same problem when you run |
|
In the mean time: I got around this by using |
|
Was this issue fixed by #178? |
|
Yes it should be. |
|
Thanks! Looking forward to a new release then :) |
|
|
I haven't looked at this in long enough of a time to remember to be sure. |
Without path configuration, coverage report shows the files from within site-packages. Also see pytest-dev/pytest-cov#130 modified: tox.ini
The code for my project, including tox.ini and .coveragerc, is at https://anonscm.debian.org/cgit/reproducible/reprotest.git/tree/?h=variations . This is on Ubuntu Linux 16.04.
I set up these tests using coverage, and now I'm trying to get subprocess coverage using pytest-cov. There's a lot of output here, unfortunately, but I'm afraid trimming it will obscure what's happening. For each run, I'm running
tox -r -- -s -k simple_builds[null]and then looking at the terminal output ofcoverage report.Running
commands = coverage run --parallel -m py.test {posargs} tests/tests.py, which is what I was using before adding pytest-cov:Normally I would run this with
--omit .tox/*also to eliminate all the non-reprotest coverage reports, but for comparison with the other reports, I left that out here. Notice that because of the [paths] section in .coveragerc, the coverage measurements for.tox/py35/lib/python3.5/site-packages/reprotest/are merged with the ones forreprotest/.With
commands = py.test --cov --cov-report=term-missing {posargs} tests/tests.py:Here, the
.tox/py35/lib/python3.5/site-packages/reprotest/measurements are not merged with the ones forreprotest/.For comparison, when I pass
--source=reprotest/tocoverage run, I get this:When I pass
--cov=reprotest/instead of--covto py.test with pytest-cov enabled, I get the same list of files, but coverage reports that it collected no data so the coverage measurements are all zero except for the empty__init__.pyfile.The text was updated successfully, but these errors were encountered: