-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add OuputVerbosity and use it in assertions #11473
Merged
Merged
Changes from all commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
c93bc1e
Add OuputVerbosity and use it in assertions
plannigan 64f7277
Add test cases to confirm implementation of mock_config()
plannigan 12cb693
Fix typo
plannigan 5b36f86
Merge branch 'main' into fine-grain-verbosity
plannigan cb9729b
Merge branch 'main' into fine-grain-verbosity
plannigan ec76cf5
Apply documentation suggestions
plannigan 933742a
Adjust verbosity level retrival
plannigan 8e38fc0
Expose VerbosityType
plannigan cfb1654
Assert isinstance intead of cast
plannigan 4d7ceb8
Improve documentation of functionality
plannigan 855f579
Merge branch 'main' into fine-grain-verbosity
plannigan d2529ed
Fix docs
nicoddemus c5cc08a
Improve docs
nicoddemus 1e91814
Remove unused _option_name_fmt
nicoddemus b8714de
Remove OutputVerbosity.verbose
nicoddemus 50c0b93
Additional design changes from review
plannigan 192b62d
Merge branch 'main' into fine-grain-verbosity
plannigan 1baca27
Update changelog/11387.feature.rst
nicoddemus 21d8111
User fine grained verbosity for saferepr max size
plannigan 675584f
Add end to end test of assertion fine grained verbosity
plannigan 9580f4b
Merge branch 'main' into fine-grain-verbosity
plannigan b47b203
More explict names
plannigan 91bfcab
Test case for unknown verbosity type
plannigan 5d2183d
Improve documentation of get_verbosity() and configuraiton
plannigan 6ea1fe5
Clean up end to end test
plannigan 8fdfc1e
Merge branch 'main' into fine-grain-verbosity
plannigan e663889
Mark constant as final
plannigan d943866
Check parser for ini name instead of manually listing known types
plannigan b91a467
Merge branch 'main' into fine-grain-verbosity
plannigan 59e39c8
Restore whitespace changes
plannigan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Added the new :confval:`verbosity_assertions` configuration option for fine-grained control of failed assertions verbosity. | ||
|
||
See :ref:`Fine-grained verbosity <pytest.fine_grained_verbosity>` for more details. | ||
|
||
For plugin authors, :attr:`config.get_verbosity <pytest.Config.get_verbosity>` can be used to retrieve the verbosity level for a specific verbosity type. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,27 +13,68 @@ | |
from _pytest import outcomes | ||
from _pytest.assertion import truncate | ||
from _pytest.assertion import util | ||
from _pytest.config import Config as _Config | ||
from _pytest.monkeypatch import MonkeyPatch | ||
from _pytest.pytester import Pytester | ||
|
||
|
||
def mock_config(verbose=0): | ||
def mock_config(verbose: int = 0, assertion_override: Optional[int] = None): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With the move away from |
||
class TerminalWriter: | ||
def _highlight(self, source, lexer): | ||
return source | ||
|
||
class Config: | ||
def getoption(self, name): | ||
if name == "verbose": | ||
return verbose | ||
raise KeyError("Not mocked out: %s" % name) | ||
|
||
def get_terminal_writer(self): | ||
return TerminalWriter() | ||
|
||
def get_verbosity(self, verbosity_type: Optional[str] = None) -> int: | ||
if verbosity_type is None: | ||
return verbose | ||
if verbosity_type == _Config.VERBOSITY_ASSERTIONS: | ||
if assertion_override is not None: | ||
return assertion_override | ||
return verbose | ||
|
||
raise KeyError(f"Not mocked out: {verbosity_type}") | ||
|
||
return Config() | ||
|
||
|
||
class TestMockConfig: | ||
SOME_VERBOSITY_LEVEL = 3 | ||
SOME_OTHER_VERBOSITY_LEVEL = 10 | ||
|
||
def test_verbose_exposes_value(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
assert config.get_verbosity() == TestMockConfig.SOME_VERBOSITY_LEVEL | ||
|
||
def test_get_assertion_override_not_set_verbose_value(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
assert ( | ||
config.get_verbosity(_Config.VERBOSITY_ASSERTIONS) | ||
== TestMockConfig.SOME_VERBOSITY_LEVEL | ||
) | ||
|
||
def test_get_assertion_override_set_custom_value(self): | ||
config = mock_config( | ||
verbose=TestMockConfig.SOME_VERBOSITY_LEVEL, | ||
assertion_override=TestMockConfig.SOME_OTHER_VERBOSITY_LEVEL, | ||
) | ||
|
||
assert ( | ||
config.get_verbosity(_Config.VERBOSITY_ASSERTIONS) | ||
== TestMockConfig.SOME_OTHER_VERBOSITY_LEVEL | ||
) | ||
|
||
def test_get_unsupported_type_error(self): | ||
config = mock_config(verbose=TestMockConfig.SOME_VERBOSITY_LEVEL) | ||
|
||
with pytest.raises(KeyError): | ||
config.get_verbosity("--- NOT A VERBOSITY LEVEL ---") | ||
bluetech marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
class TestImportHookInstallation: | ||
@pytest.mark.parametrize("initial_conftest", [True, False]) | ||
@pytest.mark.parametrize("mode", ["plain", "rewrite"]) | ||
|
@@ -1836,3 +1877,54 @@ def test_comparisons_handle_colors( | |
) | ||
|
||
result.stdout.fnmatch_lines(formatter(expected_lines), consecutive=False) | ||
|
||
|
||
def test_fine_grained_assertion_verbosity(pytester: Pytester): | ||
long_text = "Lorem ipsum dolor sit amet " * 10 | ||
p = pytester.makepyfile( | ||
f""" | ||
def test_ok(): | ||
pass | ||
|
||
|
||
def test_words_fail(): | ||
fruits1 = ["banana", "apple", "grapes", "melon", "kiwi"] | ||
fruits2 = ["banana", "apple", "orange", "melon", "kiwi"] | ||
assert fruits1 == fruits2 | ||
|
||
|
||
def test_numbers_fail(): | ||
number_to_text1 = {{str(x): x for x in range(5)}} | ||
number_to_text2 = {{str(x * 10): x * 10 for x in range(5)}} | ||
assert number_to_text1 == number_to_text2 | ||
|
||
|
||
def test_long_text_fail(): | ||
long_text = "{long_text}" | ||
assert "hello world" in long_text | ||
""" | ||
) | ||
pytester.makeini( | ||
""" | ||
[pytest] | ||
verbosity_assertions = 2 | ||
""" | ||
) | ||
result = pytester.runpytest(p) | ||
|
||
result.stdout.fnmatch_lines( | ||
[ | ||
f"{p.name} .FFF [100%]", | ||
"E At index 2 diff: 'grapes' != 'orange'", | ||
"E Full diff:", | ||
"E - ['banana', 'apple', 'orange', 'melon', 'kiwi']", | ||
"E ? ^ ^^", | ||
"E + ['banana', 'apple', 'grapes', 'melon', 'kiwi']", | ||
"E ? ^ ^ +", | ||
"E Full diff:", | ||
"E - {'0': 0, '10': 10, '20': 20, '30': 30, '40': 40}", | ||
"E ? - - - - - - - -", | ||
"E + {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4}", | ||
f"E AssertionError: assert 'hello world' in '{long_text}'", | ||
] | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we only have one aspect at the moment, it seems premature to write the prose in a general/list format.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My intention is to implement some of the other verbosity types I identified once this PR is merged. A different suggestion was to add a note that only one is currently supported. That seems sufficient to me if "life happens" and it takes me some time to get to adding other types.