-
Notifications
You must be signed in to change notification settings - Fork 685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: ssg_test_suite: warning when rule not in benchmark #10642
Conversation
ERROR message just confuses most of time and the situation is not something that can be fixed. It is lack of configuration. I think warning is better choice.
Hi @maage. Thanks for your PR. I'm waiting for a ComplianceAsCode member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Code Climate has analyzed commit 8440a09 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 52.5% (0.0% change). View more on Code Climate. |
/ok-to-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @maage, this looks reasonable. I think it will make the upstream PR reviewing process easier.
Thanks |
Description:
When rule selected to be tested is not in benchmark, issue
WARNING
, notERROR
.Rationale:
ERROR message just confuses most of time and the situation is not something that can be fixed. It is lack of configuration. I think warning is better choice.
In many cases Automatus tests fail because it checks
ERROR
string in log.For example now: #10600 there is the case.
Review Hints:
I'm not sure if this is the way to fix the issue. Maybe testing rule selection algorithm should not not select rules not in benchmarks. Warnings are still visible in logs.