Rework of `cpe_generate.py` #11644

Honny1 · 2024-03-01T19:00:01Z

Description:

This PR reworks the cpe_generate.py script. It simplifies the code for future enhancements such as reducing CPE AL in the data stream. The script uses the OVAL object model to process OVAL. And it separates the process of creating the CPE dictionary and the CPE OVAL.

Review Hints:

These changes shouldn't change the functionality of the script.

openshift-ci · 2024-03-01T19:00:11Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

github-actions · 2024-03-01T19:01:28Z

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

github-actions · 2024-03-01T19:05:17Z

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:11644
This image was built from commit: dbf1e69

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:11644

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:11644 make deploy-local

jan-cerny · 2024-03-06T11:28:38Z

build-scripts/cpe_generate.py

+    # Lets scrape the shorthand for the list of platforms referenced
+    benchmark_cpe_names = get_benchmark_cpe_names(args.shorthandfile)


This file actually isn't in shorthand format. It's build/${product}/ssg-${product}-xccdf.xml which is a normal XCCDF document. This comment is outdated. We should change the comment and rename all variables accordingly. Also, we need to rename the argument shorthandfile.

jan-cerny · 2024-03-06T11:31:39Z

build-scripts/cpe_generate.py

        # skip CPE AL platforms (they are handled later)
        # this is temporary solution until we get rid of old type of platforms in the benchmark
        if cpe_name.startswith("#"):


Can you check if what the comment says is still the case?

Yes, I think it's still relevant because, for example, OS platforms use CPE identifiers and CPE dictionaries.

jan-cerny · 2024-03-06T11:52:11Z

build-scripts/cpe_generate.py

-
-    # parse oval file
-    ovaltree = ssg.xml.parse_file(args.ovalfile)
+def load_oval(args):


It's better to not pass args down here. Instead, this function should have 2 specific arguments.

jan-cerny · 2024-03-06T11:54:20Z

build-scripts/cpe_generate.py

-
-    # parse oval file
-    ovaltree = ssg.xml.parse_file(args.ovalfile)
+def load_oval(args):


I think the name of this function is misleading, it's load_oval but it actually does many more actions that just loading. Please rename it.

jan-cerny · 2024-03-06T12:03:13Z

build-scripts/cpe_generate.py

    # turn IDs into meaningless numbers
    translator = ssg.id_translate.IDTranslator(args.idname)
-    ovaltree = translator.translate(ovaltree)
+    oval_document = translator.translate_oval_document(oval_document, store_defname=True)


The IDs aren't turned into "meaningless numbers", they're added prefix, suffix and number, which makes them unique, but not meaningless.

jan-cerny · 2024-03-06T12:16:38Z

build-scripts/cpe_generate.py


+def load_cpe_dictionary(benchmark_cpe_names, product_yaml, args):


I don't like that the args is passed down in here. I think that the function should have 2 specific arguments instead of the args argument.

jan-cerny · 2024-03-06T12:17:48Z

build-scripts/cpe_generate.py

+    p.add_argument(
+        "idname",
+        help="Identifier prefix"
+    )


When we are here can we actually make this a constant please? I think that it's unlikely to ever be anything different than ssg.

jan-cerny · 2024-03-06T13:20:41Z

Also, I have noticed that there are frequent typos in your commit messages. I think I pointed out this before in one of your previous PRs. Please employ a spell checker.

codeclimate · 2024-03-07T11:19:53Z

Code Climate has analyzed commit dbf1e69 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 50.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.8% (0.0% change).

View more on Code Climate.

jan-cerny

I have reviewed the code and I have reviewed the generated output from the RHEL 8 product, focusing on the CPE OVAL and CPE dictionary.

openshift-ci bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Mar 1, 2024

Honny1 force-pushed the rework-cpe-generate branch from 4ee07e0 to aef3461 Compare March 4, 2024 13:13

Honny1 marked this pull request as ready for review March 4, 2024 14:01

openshift-ci bot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Mar 4, 2024

Honny1 added enhancement General enhancements to the project. Infrastructure Our content build system labels Mar 5, 2024

Honny1 mentioned this pull request Mar 5, 2024

Reduction of CPE content in DS #11648

Merged

jan-cerny self-assigned this Mar 6, 2024

jan-cerny reviewed Mar 6, 2024

View reviewed changes

Honny1 added 7 commits March 7, 2024 11:30

Format argument parsing

afd121f

Use OVAL object model to generate linked CPE OVAL

d5f8f90

Decompose CPE OVAL and CPE dictionary generation

ba646a3

Translation of IDs to valid OVAL IDs

294d93f

Remove unused ID checks and translations

3a64a81

Use format string

48938d0

Use base name of file

36cb638

Honny1 force-pushed the rework-cpe-generate branch from 5cc08ba to eede727 Compare March 7, 2024 10:59

Honny1 added 5 commits March 7, 2024 12:02

Rename shorthandfile to xccdfFile

a1e11f9

Remove identifier prefix argument

de57662

Pass to functions specific arguments instead args

5e4f72e

Rename function load_oval to process_cpe_oval

66549d1

Remove outdated comments

dbf1e69

Honny1 force-pushed the rework-cpe-generate branch from eede727 to dbf1e69 Compare March 7, 2024 11:05

jan-cerny approved these changes Mar 7, 2024

View reviewed changes

jan-cerny merged commit f0474eb into ComplianceAsCode:master Mar 7, 2024
44 checks passed

jan-cerny added this to the 0.1.73 milestone Mar 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework of `cpe_generate.py` #11644

Rework of `cpe_generate.py` #11644

Honny1 commented Mar 1, 2024 •

edited

Loading

openshift-ci bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024 •

edited

Loading

jan-cerny Mar 6, 2024

jan-cerny Mar 6, 2024

Honny1 Mar 7, 2024

jan-cerny Mar 6, 2024

jan-cerny Mar 6, 2024

jan-cerny Mar 6, 2024

jan-cerny Mar 6, 2024

jan-cerny Mar 6, 2024

jan-cerny commented Mar 6, 2024

codeclimate bot commented Mar 7, 2024

jan-cerny left a comment

		# Lets scrape the shorthand for the list of platforms referenced
		benchmark_cpe_names = get_benchmark_cpe_names(args.shorthandfile)


		def load_cpe_dictionary(benchmark_cpe_names, product_yaml, args):

Rework of cpe_generate.py #11644

Rework of cpe_generate.py #11644

Conversation

Honny1 commented Mar 1, 2024 • edited Loading

Description:

Review Hints:

openshift-ci bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-cerny commented Mar 6, 2024

codeclimate bot commented Mar 7, 2024

jan-cerny left a comment

Choose a reason for hiding this comment

Rework of `cpe_generate.py` #11644

Rework of `cpe_generate.py` #11644

Honny1 commented Mar 1, 2024 •

edited

Loading

github-actions bot commented Mar 1, 2024 •

edited

Loading