Reduction of CPE content in DS #11648

Honny1 · 2024-03-05T12:49:50Z

Description:

This PR reduces the CPE OVAL, CPE AL and CPE dictionary in the data stream. This PR also includes the integration of minimal CPE components into thin DS. For example, the data stream size reduction generated by ./build_product fedora --rule-id audit_rules_privileged_commands_fusermount3 is 249K.

This PR also deals with the problem of adding new CPE platforms to the derivative. In connection with this, an automatically formatted derivative is generated.

Review Hints:

To test one rule you can run this command:

./build_product fedora --rule-id audit_rules_privileged_commands_fusermount3

To test changes you can run this script:

#!/bin/bash

./build_product fedora --thin

for filename in ./build/thin_ds/*.xml; do
    echo "$filename"
    oscap xccdf eval  --profile "(all)" --results-arf "arf_results/arf_$(basename -- $filename)" "$filename"
done

The script generates a thin Datastream for each rule and then performs a scan using oscap.
This test takes more than an hour to run because there are approximately 1830 rules to process and some are memory intensive.

Depends on:

Rework of cpe_generate.py #11644

openshift-ci · 2024-03-05T12:50:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

github-actions · 2024-03-05T12:51:16Z

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

github-actions · 2024-03-05T12:55:17Z

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:11648
This image was built from commit: 2647ff4

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:11648

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:11648 make deploy-local

jan-cerny

This is a great improvement!

What are the new added reference elements in the CPE OVAL definitions? They aren't there in current master. For example: <oval-def:reference ref_id="installed_env_is_a_container" source="ssg"/>

Honny1 · 2024-03-07T22:28:29Z

/packit retest-failed

Honny1 · 2024-03-07T22:31:28Z

@jan-cerny This new reference is created during translation to map the original id before translation. Do we want to keep it? It can be easily turned off.

jan-cerny · 2024-03-08T07:17:14Z

@Honny1 I'd propose to turn it off, I don't know what value it brings.

jan-cerny · 2024-03-08T07:39:25Z

ssg/build_cpe.py


+        if hasattr(ET, "indent"):
+            ET.indent(cpe_list, space=" ", level=0)


Usually 2 spaces are used in XMLs.

jan-cerny · 2024-03-08T07:43:03Z

build-scripts/cpe_generate.py

@@ -22,6 +23,7 @@

 oval_ns = "http://oval.mitre.org/XMLSchema/oval-definitions-5"
 cpe_ns = "http://cpe.mitre.org/dictionary/2.0"
+cpe_lang_ns = ssg.constants.PREFIX_TO_NS["cpe-lang"]


I know that this isn't done by you but I think you should move all these namespace definitions to ssg/constants.py and only import it from there here. Also, the cpe-lang NS should be defined as a separate constant and not using the PREFIX_TO_NS dictionary which should only reference the constant.

jan-cerny · 2024-03-08T14:53:23Z

So far it looks great. I have seen the thin data streams. I also have seen that the unused CPE items and the corresponding unused CPE OVAL definitions are succesfully removed also from the classic data stream which helps reduce it size.

However, the CI tail of TF on CS8 is a legit failure and needs to be fixed fixed. The problem happens with the derivative products, such as CentOS 8. The derivative data stream ssg-centos8-ds.xml doesn't contain the OVAL definition installed_OS_is_centos8 for the platform .cpe:/o:centos:centos:8.

This problem is most likely caused by the fact that your code now strips off all the unused items, but when builiding derivatives, new items are added later by the enable_derivatives.py. You can see that this scripts adds additional CPEs for CS or SL.

You need to invent how to workaround this problem. Maybe add a special case for the definitions used in the derivatives data streams so that they are never removed?

Honny1 · 2024-03-13T12:11:58Z

/packit retest-failed

jan-cerny · 2024-03-13T14:17:58Z

/packit retest-failed

jan-cerny · 2024-03-13T15:15:21Z

cmake/SSGCommon.cmake

@@ -944,8 +944,7 @@ macro(ssg_build_derivative_product ORIGINAL SHORTNAME DERIVATIVE)

    add_custom_command(
        OUTPUT "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml"
-        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/enable_derivatives.py" --enable-${SHORTNAME} -i "${CMAKE_BINARY_DIR}/ssg-${ORIGINAL}-ds.xml" -o "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_CURRENT_BINARY_DIR}/product.yml" ${DERIVATIVE} --id-name ssg --cpe-items-dir "${CMAKE_CURRENT_BINARY_DIR}/cpe_items"
-        COMMAND "${XMLLINT_EXECUTABLE}" --nsclean --format --output "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml"
+        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/enable_derivatives.py" --enable-${SHORTNAME} -i "${CMAKE_BINARY_DIR}/ssg-${ORIGINAL}-ds.xml" -o "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_CURRENT_BINARY_DIR}/product.yml" ${DERIVATIVE} --id-name ssg --cpe-items-dir "${CMAKE_CURRENT_BINARY_DIR}/cpe_items" --unlinked-cpe-oval-path "${CMAKE_CURRENT_BINARY_DIR}/cpe-oval-unlinked.xml"


I think that the xmllint reformat needs to stay there because apparently OpenSCAP has problems with not pretty formatted XMLs and on RHEL 7 the Python doesn't produce pretty outputs.

The segfault is reproducible, it's not a random glitch.

Ok, fixed, after RHEL 7 is not supported we can remove the formatting tool.

codeclimate · 2024-03-14T12:05:06Z

Code Climate has analyzed commit 2647ff4 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 13.7% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.3% (-0.5% change).

View more on Code Climate.

Honny1 · 2024-03-14T13:01:18Z

/packit retest-failed

Honny1 · 2024-03-14T14:06:28Z

/packit retest-failed

jan-cerny

Today I reviewed the code again. I have built Fedora and RHEL 8 products and checked some of the thin data streams and also single rule data streams and tried to evaluated them. I also have built derivative products and checked that the CPE entry has been inserted there.

openshift-ci bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Mar 5, 2024

Honny1 force-pushed the strip-cpe branch from 5e6c799 to 4657701 Compare March 5, 2024 13:50

jan-cerny self-assigned this Mar 7, 2024

jan-cerny added Infrastructure Our content build system CPE-AL CPE Applicability Language labels Mar 7, 2024

jan-cerny reviewed Mar 7, 2024

View reviewed changes

Honny1 force-pushed the strip-cpe branch 2 times, most recently from 6252bed to 2fd9f84 Compare March 7, 2024 16:32

Honny1 marked this pull request as ready for review March 7, 2024 22:28

openshift-ci bot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Mar 7, 2024

jan-cerny added this to the 0.1.73 milestone Mar 8, 2024

jan-cerny reviewed Mar 8, 2024

View reviewed changes

Honny1 force-pushed the strip-cpe branch from d111aac to 33cde89 Compare March 8, 2024 13:25

Honny1 force-pushed the strip-cpe branch 4 times, most recently from 3f10a14 to 6afbbe8 Compare March 12, 2024 14:00

jan-cerny requested changes Mar 13, 2024

View reviewed changes

Honny1 added 3 commits March 14, 2024 12:50

Get used and not used cpe platforms

ab5d0b8

Disable generation of not used cpe platform

8644d7c

Get used IDs of CPE OVAL definitions

52c4646

Honny1 added 13 commits March 14, 2024 12:50

Add filtration of OVAL document

e00bcae

Add filtration of CPE to generation of CPE dict XML

5fd49fc

Fix bug when parameter is not present

b4dd938

Add generation of CPE dict and CPE OVAL for thin DS

299046b

Add CPE dict and CPE OVAL to final thin DS

4c73910

Fix CPE OVAL references

92100c4

Reduce duplication

90b211d

Turn off the reference element in CPE OVAL

97bae48

Use namesapces from constants

1525db4

Create getter for linked CPE OVAL

184cf3a

Create functions to insert new CPE OVAL definitions

42bce2d

Add CPE OVAL for new CPE platforms

f762854

Generate formatted derivates

2647ff4

Honny1 force-pushed the strip-cpe branch from 6afbbe8 to 2647ff4 Compare March 14, 2024 11:50

Honny1 requested a review from jan-cerny March 14, 2024 15:07

jan-cerny approved these changes Mar 15, 2024

View reviewed changes

jan-cerny merged commit 0cfee04 into ComplianceAsCode:master Mar 15, 2024
42 of 44 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduction of CPE content in DS #11648

Reduction of CPE content in DS #11648

Honny1 commented Mar 5, 2024 •

edited

Loading

openshift-ci bot commented Mar 5, 2024

github-actions bot commented Mar 5, 2024

github-actions bot commented Mar 5, 2024 •

edited

Loading

jan-cerny left a comment

Honny1 commented Mar 7, 2024

Honny1 commented Mar 7, 2024 •

edited

Loading

jan-cerny commented Mar 8, 2024

jan-cerny Mar 8, 2024

jan-cerny Mar 8, 2024

jan-cerny commented Mar 8, 2024

Honny1 commented Mar 13, 2024

jan-cerny commented Mar 13, 2024

jan-cerny Mar 13, 2024

Honny1 Mar 14, 2024

codeclimate bot commented Mar 14, 2024

Honny1 commented Mar 14, 2024

Honny1 commented Mar 14, 2024

jan-cerny left a comment


		if hasattr(ET, "indent"):
		ET.indent(cpe_list, space=" ", level=0)

Reduction of CPE content in DS #11648

Reduction of CPE content in DS #11648

Conversation

Honny1 commented Mar 5, 2024 • edited Loading

Description:

Review Hints:

openshift-ci bot commented Mar 5, 2024

github-actions bot commented Mar 5, 2024

github-actions bot commented Mar 5, 2024 • edited Loading

jan-cerny left a comment

Choose a reason for hiding this comment

Honny1 commented Mar 7, 2024

Honny1 commented Mar 7, 2024 • edited Loading

jan-cerny commented Mar 8, 2024

jan-cerny Mar 8, 2024

Choose a reason for hiding this comment

jan-cerny Mar 8, 2024

Choose a reason for hiding this comment

jan-cerny commented Mar 8, 2024

Honny1 commented Mar 13, 2024

jan-cerny commented Mar 13, 2024

jan-cerny Mar 13, 2024

Choose a reason for hiding this comment

Honny1 Mar 14, 2024

Choose a reason for hiding this comment

codeclimate bot commented Mar 14, 2024

Honny1 commented Mar 14, 2024

Honny1 commented Mar 14, 2024

jan-cerny left a comment

Choose a reason for hiding this comment

Honny1 commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading

Honny1 commented Mar 7, 2024 •

edited

Loading