You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Automatus test scenarios for rule set_nftables_table fail when they are executed as a part of daily productization tests /per-rule/12/ansible and /per-rule/12/oscap. They give notchecked result instead of the expected results.
The reason is that the rule has only an SCE check and doesn't have an OVAL check. We don't build the content with SCE. SCE isn't included by default. Therefore, the notechecked is expected.
But the problem is how to handle this situation. We can waive it permanently, or we can introduce some logic for this situation, for example we can add a special test scenarios header.
SCAP Security Guide Version:
current upstream master as of 2024-05-27 as of HEAD 2f2a8c7
Operating System Version:
RHEL-9.4.0-20240526.24
Steps to Reproduce:
this is run inside contest
./automatus.py rule --libvirt qemu:///system contest --product rhel9 (--remediate-using ansible) ... set_nftables_table ...
Actual Results:
INFO - xccdf_org.ssgproject.content_rule_set_nftables_table
2024-05-28 09:47:41 test.py:126: running for rule: set_nftables_table
ERROR - Script nftables_incorrect_family.fail.sh using profile (all) found issue:
2024-05-28 09:47:47 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_incorrect_family.fail [/tmp/tmpgbnqb2tv/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_no_tables.fail.sh using profile (all) found issue:
2024-05-28 09:47:56 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_no_tables.fail [/tmp/tmpgb_3auju/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_table_present.pass.sh using profile (all) found issue:
2024-05-28 09:48:05 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_table_present.pass [/tmp/tmpiylbmqjp/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected pass during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
Expected Results:
There shouldn't be any error. One of the tasks that we need to do is to determine what should be the expected results for this rule in this situation.
We introduce a new test scenarios header `check` that allows
to mark test scenarios as specific to a single check engine
type. For example, adding header `check = sce` to a test scenario
marks this test scenario as specific only to SCE. If SCE check
isn't available, such scenario will be skipped. If the `check`
header isn't specified in a test scenario, the test scenario
will work with any check type.
Fixes: ComplianceAsCode#12030
Description of problem:
Automatus test scenarios for rule
set_nftables_table
fail when they are executed as a part of daily productization tests/per-rule/12/ansible
and/per-rule/12/oscap
. They givenotchecked
result instead of the expected results.The reason is that the rule has only an SCE check and doesn't have an OVAL check. We don't build the content with SCE. SCE isn't included by default. Therefore, the
notechecked
is expected.But the problem is how to handle this situation. We can waive it permanently, or we can introduce some logic for this situation, for example we can add a special test scenarios header.
SCAP Security Guide Version:
current upstream master as of 2024-05-27 as of HEAD 2f2a8c7
Operating System Version:
RHEL-9.4.0-20240526.24
Steps to Reproduce:
this is run inside contest
./automatus.py rule --libvirt qemu:///system contest --product rhel9 (--remediate-using ansible) ... set_nftables_table ...
Actual Results:
Expected Results:
There shouldn't be any error. One of the tasks that we need to do is to determine what should be the expected results for this rule in this situation.
Additional Information/Debugging Steps:
The tests have been added by #11991.
The text was updated successfully, but these errors were encountered: