-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add OuputVerbosity and use it in assertions #11473
Changes from 21 commits
c93bc1e
64f7277
12cb693
5b36f86
cb9729b
ec76cf5
933742a
8e38fc0
cfb1654
4d7ceb8
855f579
d2529ed
c5cc08a
1e91814
b8714de
50c0b93
192b62d
1baca27
21d8111
675584f
9580f4b
b47b203
91bfcab
5d2183d
6ea1fe5
8fdfc1e
e663889
d943866
b91a467
59e39c8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Added the new :confval:`verbosity_assertions` configuration option for fine-grained control of failed assertions verbosity. | ||
|
||
See :ref:`Fine-grained verbosity <pytest.fine_grained_verbosity>` for more details. | ||
|
||
For plugin authors, :attr:`config.get_verbosity <pytest.Config.get_verbosity>` can be used to retrieve the verbosity level for a specific verbosity type. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,6 +22,7 @@ | |
from typing import Callable | ||
from typing import cast | ||
from typing import Dict | ||
from typing import Final | ||
from typing import final | ||
from typing import Generator | ||
from typing import IO | ||
|
@@ -69,7 +70,7 @@ | |
if TYPE_CHECKING: | ||
from _pytest._code.code import _TracebackStyle | ||
from _pytest.terminal import TerminalReporter | ||
from .argparsing import Argument | ||
from .argparsing import Argument, Parser | ||
|
||
|
||
_PluggyPlugin = object | ||
|
@@ -1650,6 +1651,66 @@ | |
"""Deprecated, use getoption(skip=True) instead.""" | ||
return self.getoption(name, skip=True) | ||
|
||
#: Verbosity for failed assertions (see :confval:`verbosity_assertions`). | ||
VERBOSITY_ASSERTIONS: Final = "assertions" | ||
_KNOWN_VERBOSITY_TYPES: Final = {VERBOSITY_ASSERTIONS} | ||
_VERBOSITY_INI_DEFAULT = "auto" | ||
plannigan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
def get_verbosity(self, verbosity_type: Optional[str] = None) -> int: | ||
plannigan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
r"""Access to fine-grained verbosity levels. | ||
|
||
.. code-block:: ini | ||
|
||
# content of pytest.ini | ||
[pytest] | ||
verbosity_assertions = 2 | ||
|
||
.. code-block:: console | ||
|
||
pytest -v | ||
|
||
.. code-block:: python | ||
|
||
print(config.get_verbosity()) # 1 | ||
print(config.get_verbosity(Config.VERBOSITY_ASSERTIONS)) # 2 | ||
""" | ||
global_level = self.option.verbose | ||
assert isinstance(global_level, int) | ||
if ( | ||
verbosity_type is None | ||
or verbosity_type not in Config._KNOWN_VERBOSITY_TYPES | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be better to avoid this check, so that plugins can add their own verbosity types. We would instead check if the ini exists. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I like the idea of checking the ini data instead (one less thing to manually manage). In a previous conversation, there was a conversation about plugins creating their own levels. I think I'm still in the "pytest can enumerate a few buckets" camp. @nicoddemus Do you have any thoughts? If we did allow plugins to create types, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can keep it simple for now, making this internal only; if the need comes later, we can discuss how to make this public -- better to err on the side of caution, as it is easy to make something public later, rather than making something public and then regretting it later. |
||
): | ||
return global_level | ||
plannigan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
level = self.getini(Config._ini_name(verbosity_type)) | ||
|
||
if level == Config._VERBOSITY_INI_DEFAULT: | ||
bluetech marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return global_level | ||
|
||
return int(level) | ||
|
||
@staticmethod | ||
bluetech marked this conversation as resolved.
Show resolved
Hide resolved
|
||
def _ini_name(verbosity_type: str) -> str: | ||
return f"verbosity_{verbosity_type}" | ||
|
||
@staticmethod | ||
def _add_ini(parser: "Parser", verbosity_type: str, help: str) -> None: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here, the name Also, IMO this function doesn't add much value, I think it would be better to reduce the indirection and inline it. But since it's private then it's OK if you want to keep it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I prefer to keep it because it will avoid duplication as more types are added. It also ensures that the same default is used across verbosity types. If someone adds an Nth type, but uses "global", then |
||
"""Add a output verbosity configuration option for the given output type. | ||
|
||
:param parser: Parser for command line arguments and ini-file values. | ||
:param verbosity_type: Fine-grained verbosity category. | ||
:param help: Description of the output this type controls. | ||
|
||
The value should be retrieved via a call to | ||
:py:func:`config.get_verbosity(type) <pytest.Config.get_verbosity>`. | ||
""" | ||
parser.addini( | ||
Config._ini_name(verbosity_type), | ||
help=help, | ||
type="string", | ||
default=Config._VERBOSITY_INI_DEFAULT, | ||
) | ||
|
||
def _warn_about_missing_assertion(self, mode: str) -> None: | ||
if not _assertion_supported(): | ||
if mode == "plain": | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,27 +13,68 @@ | |
from _pytest import outcomes | ||
from _pytest.assertion import truncate | ||
from _pytest.assertion import util | ||
from _pytest.config import Config as _Config | ||
from _pytest.monkeypatch import MonkeyPatch | ||
from _pytest.pytester import Pytester | ||
|
||
|
||
def mock_config(verbose=0): | ||
def mock_config(verbose: int = 0, assertion_override: Optional[int] = None): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With the move away from |
||
class TerminalWriter: | ||
def _highlight(self, source, lexer): | ||
return source | ||
|
||
class Config: | ||
def getoption(self, name): | ||
if name == "verbose": | ||
return verbose | ||
raise KeyError("Not mocked out: %s" % name) | ||
|
||
def get_terminal_writer(self): | ||
return TerminalWriter() | ||
|
||
def get_verbosity(self, verbosity_type: Optional[str] = None) -> int: | ||
if verbosity_type is None: | ||
return verbose | ||
if verbosity_type == _Config.VERBOSITY_ASSERTIONS: | ||
if assertion_override is not None: | ||
return assertion_override | ||
return verbose | ||
|
||
raise KeyError(f"Not mocked out: {verbosity_type}") | ||
|
||
return Config() | ||
|
||
|
||
class TestMockConfig: | ||
SOME_VERBOSITY_LEVEL = 3 | ||
SOME_OTHER_VERBOSITY_LEVEL = 10 | ||
|
||
def test_verbose_exposes_value(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
assert config.get_verbosity() == TestMockConfig.SOME_VERBOSITY_LEVEL | ||
|
||
def test_get_assertion_override_not_set_verbose_value(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
assert ( | ||
config.get_verbosity(_Config.VERBOSITY_ASSERTIONS) | ||
== TestMockConfig.SOME_VERBOSITY_LEVEL | ||
) | ||
|
||
def test_get_assertion_override_set_custom_value(self): | ||
config = mock_config( | ||
verbose=TestMockConfig.SOME_VERBOSITY_LEVEL, | ||
assertion_override=TestMockConfig.SOME_OTHER_VERBOSITY_LEVEL, | ||
) | ||
|
||
assert ( | ||
config.get_verbosity(_Config.VERBOSITY_ASSERTIONS) | ||
== TestMockConfig.SOME_OTHER_VERBOSITY_LEVEL | ||
) | ||
|
||
def test_get_unsupported_type_error(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
with pytest.raises(KeyError): | ||
config.get_verbosity("--- NOT A VERBOSITY LEVEL ---") | ||
bluetech marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
class TestImportHookInstallation: | ||
@pytest.mark.parametrize("initial_conftest", [True, False]) | ||
@pytest.mark.parametrize("mode", ["plain", "rewrite"]) | ||
|
@@ -1836,3 +1877,54 @@ def test_comparisons_handle_colors( | |
) | ||
|
||
result.stdout.fnmatch_lines(formatter(expected_lines), consecutive=False) | ||
|
||
|
||
def test_fine_grained_assertion_verbosity(pytester: Pytester): | ||
long_text = "Lorem ipsum dolor sit amet " * 10 | ||
p = pytester.makepyfile( | ||
f""" | ||
def test_ok(): | ||
pass | ||
|
||
|
||
def test_words_fail(): | ||
fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"] | ||
fruits2 = ["banana", "apple", "orange", "melon", "kiwi"] | ||
assert fruits1 == fruits2 | ||
|
||
|
||
def test_numbers_fail(): | ||
number_to_text1 = {{str(x): x for x in range(5)}} | ||
number_to_text2 = {{str(x * 10): x * 10 for x in range(5)}} | ||
assert number_to_text1 == number_to_text2 | ||
|
||
|
||
def test_long_text_fail(): | ||
long_text = "{long_text}" | ||
assert "hello world" in long_text | ||
""" | ||
) | ||
pytester.makeini( | ||
f""" | ||
[pytest] | ||
{_Config._ini_name(_Config.VERBOSITY_ASSERTIONS)} = 2 | ||
plannigan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
""" | ||
) | ||
result = pytester.runpytest(str(p)) | ||
plannigan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
result.stdout.fnmatch_lines( | ||
[ | ||
f"{p.name} .FFF [100%]", | ||
"E At index 2 diff: 'grapes' != 'orange'", | ||
"E Full diff:", | ||
"E - ['banana', 'apple', 'orange', 'melon', 'kiwi']", | ||
"E ? ^ ^^", | ||
"E + ['banana', 'apple', 'grapes', 'melon', 'kiwi']", | ||
"E ? ^ ^ +", | ||
"E Full diff:", | ||
"E - {'0': 0, '10': 10, '20': 20, '30': 30, '40': 40}", | ||
"E ? - - - - - - - -", | ||
"E + {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4}", | ||
f"E AssertionError: assert 'hello world' in '{long_text}'", | ||
] | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we only have one aspect at the moment, it seems premature to write the prose in a general/list format.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My intention is to implement some of the other verbosity types I identified once this PR is merged. A different suggestion was to add a note that only one is currently supported. That seems sufficient to me if "life happens" and it takes me some time to get to adding other types.