-
Notifications
You must be signed in to change notification settings - Fork 685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduction of CPE content in DS #11648
Conversation
Skipping CI for Draft Pull Request. |
🤖 A k8s content image for this PR is available at: Click here to see how to deploy itIf you alread have Compliance Operator deployed: Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great improvement!
What are the new added reference elements in the CPE OVAL definitions? They aren't there in current master. For example: <oval-def:reference ref_id="installed_env_is_a_container" source="ssg"/>
6252bed
to
2fd9f84
Compare
/packit retest-failed |
@jan-cerny This new reference is created during translation to map the original id before translation. Do we want to keep it? It can be easily turned off. |
@Honny1 I'd propose to turn it off, I don't know what value it brings. |
ssg/build_cpe.py
Outdated
|
||
if hasattr(ET, "indent"): | ||
ET.indent(cpe_list, space=" ", level=0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usually 2 spaces are used in XMLs.
build-scripts/cpe_generate.py
Outdated
@@ -22,6 +23,7 @@ | |||
|
|||
oval_ns = "http://oval.mitre.org/XMLSchema/oval-definitions-5" | |||
cpe_ns = "http://cpe.mitre.org/dictionary/2.0" | |||
cpe_lang_ns = ssg.constants.PREFIX_TO_NS["cpe-lang"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know that this isn't done by you but I think you should move all these namespace definitions to ssg/constants.py
and only import it from there here. Also, the cpe-lang
NS should be defined as a separate constant and not using the PREFIX_TO_NS
dictionary which should only reference the constant.
So far it looks great. I have seen the thin data streams. I also have seen that the unused CPE items and the corresponding unused CPE OVAL definitions are succesfully removed also from the classic data stream which helps reduce it size. However, the CI tail of TF on CS8 is a legit failure and needs to be fixed fixed. The problem happens with the derivative products, such as CentOS 8. The derivative data stream This problem is most likely caused by the fact that your code now strips off all the unused items, but when builiding derivatives, new items are added later by the You need to invent how to workaround this problem. Maybe add a special case for the definitions used in the derivatives data streams so that they are never removed? |
3f10a14
to
6afbbe8
Compare
/packit retest-failed |
1 similar comment
/packit retest-failed |
@@ -944,8 +944,7 @@ macro(ssg_build_derivative_product ORIGINAL SHORTNAME DERIVATIVE) | |||
|
|||
add_custom_command( | |||
OUTPUT "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" | |||
COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/enable_derivatives.py" --enable-${SHORTNAME} -i "${CMAKE_BINARY_DIR}/ssg-${ORIGINAL}-ds.xml" -o "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_CURRENT_BINARY_DIR}/product.yml" ${DERIVATIVE} --id-name ssg --cpe-items-dir "${CMAKE_CURRENT_BINARY_DIR}/cpe_items" | |||
COMMAND "${XMLLINT_EXECUTABLE}" --nsclean --format --output "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" | |||
COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/enable_derivatives.py" --enable-${SHORTNAME} -i "${CMAKE_BINARY_DIR}/ssg-${ORIGINAL}-ds.xml" -o "${CMAKE_BINARY_DIR}/ssg-${DERIVATIVE}-ds.xml" "${CMAKE_CURRENT_BINARY_DIR}/product.yml" ${DERIVATIVE} --id-name ssg --cpe-items-dir "${CMAKE_CURRENT_BINARY_DIR}/cpe_items" --unlinked-cpe-oval-path "${CMAKE_CURRENT_BINARY_DIR}/cpe-oval-unlinked.xml" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that the xmllint reformat needs to stay there because apparently OpenSCAP has problems with not pretty formatted XMLs and on RHEL 7 the Python doesn't produce pretty outputs.
The segfault is reproducible, it's not a random glitch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, fixed, after RHEL 7 is not supported we can remove the formatting tool.
Code Climate has analyzed commit 2647ff4 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 13.7% (50% is the threshold). This pull request will bring the total coverage in the repository to 59.3% (-0.5% change). View more on Code Climate. |
/packit retest-failed |
1 similar comment
/packit retest-failed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Today I reviewed the code again. I have built Fedora and RHEL 8 products and checked some of the thin data streams and also single rule data streams and tried to evaluated them. I also have built derivative products and checked that the CPE entry has been inserted there.
Description:
This PR reduces the CPE OVAL, CPE AL and CPE dictionary in the data stream. This PR also includes the integration of minimal CPE components into thin DS. For example, the data stream size reduction generated by
./build_product fedora --rule-id audit_rules_privileged_commands_fusermount3
is 249K.This PR also deals with the problem of adding new CPE platforms to the derivative. In connection with this, an automatically formatted derivative is generated.
Review Hints:
To test one rule you can run this command:
To test changes you can run this script:
The script generates a thin Datastream for each rule and then performs a scan using
oscap
.This test takes more than an hour to run because there are approximately 1830 rules to process and some are memory intensive.
Depends on:
cpe_generate.py
#11644