Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test scenarios fail for SCE-only rules if built without SCE #12030

Closed
jan-cerny opened this issue May 28, 2024 · 1 comment · Fixed by #12052
Closed

Test scenarios fail for SCE-only rules if built without SCE #12030

jan-cerny opened this issue May 28, 2024 · 1 comment · Fixed by #12052
Assignees
Labels
productization-issue Issue found in upstream stabilization process.
Milestone

Comments

@jan-cerny
Copy link
Collaborator

Description of problem:

Automatus test scenarios for rule set_nftables_table fail when they are executed as a part of daily productization tests /per-rule/12/ansible and /per-rule/12/oscap. They give notchecked result instead of the expected results.

The reason is that the rule has only an SCE check and doesn't have an OVAL check. We don't build the content with SCE. SCE isn't included by default. Therefore, the notechecked is expected.

But the problem is how to handle this situation. We can waive it permanently, or we can introduce some logic for this situation, for example we can add a special test scenarios header.

SCAP Security Guide Version:

current upstream master as of 2024-05-27 as of HEAD 2f2a8c7

Operating System Version:

RHEL-9.4.0-20240526.24

Steps to Reproduce:

this is run inside contest
./automatus.py rule --libvirt qemu:///system contest --product rhel9 (--remediate-using ansible) ... set_nftables_table ...

Actual Results:

INFO - xccdf_org.ssgproject.content_rule_set_nftables_table
2024-05-28 09:47:41 test.py:126: running for rule: set_nftables_table
ERROR - Script nftables_incorrect_family.fail.sh using profile (all) found issue:
2024-05-28 09:47:47 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_incorrect_family.fail [/tmp/tmpgbnqb2tv/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_no_tables.fail.sh using profile (all) found issue:
2024-05-28 09:47:56 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_no_tables.fail [/tmp/tmpgb_3auju/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_table_present.pass.sh using profile (all) found issue:
2024-05-28 09:48:05 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_table_present.pass [/tmp/tmpiylbmqjp/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected pass during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.

Expected Results:

There shouldn't be any error. One of the tasks that we need to do is to determine what should be the expected results for this rule in this situation.

Additional Information/Debugging Steps:

The tests have been added by #11991.

@jan-cerny jan-cerny added the productization-issue Issue found in upstream stabilization process. label May 28, 2024
@jan-cerny jan-cerny added this to the 0.1.74 milestone May 28, 2024
@jan-cerny
Copy link
Collaborator Author

adding to contest as a waiver: RHSecurityCompliance/contest#195

jan-cerny added a commit to jan-cerny/scap-security-guide that referenced this issue Jun 7, 2024
We introduce a new test scenarios header `check` that allows
to mark test scenarios as specific to a single check engine
type. For example, adding header `check = sce` to a test scenario
marks this test scenario as specific only to SCE. If SCE check
isn't available, such scenario will be skipped. If the `check`
header isn't specified in a test scenario, the test scenario
will work with any check type.

Fixes: ComplianceAsCode#12030
@jan-cerny jan-cerny self-assigned this Jun 7, 2024
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
productization-issue Issue found in upstream stabilization process.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant