Do not instantiate Builder() when running Automatus

[yuumasato]

Description:

• Remove functions that fetch test scenario paths and load test scenarios from the Builder() class.
  • These functions are not used to build templated content.
  • This removes the need to intantiate a mock Builder() just for the sake of getting the test scenarios and their paths.
• Do not try to generate ProductCPEs() when platform_dirs or cpe_items_dir are not provided.

Rationale:

• Automatus doesn't need to instantiate Builder() to run the test scenarios.
• Automatus doesn't need to generate ProductCPEs() to run test scenarios, they are only used when building rules into data stream.
• This avoids need to provide platform-dirs and cpe-items-dir when Automatus doesn't use ProductCPEs() for anything.
• Fixes Automatus CI TypeError: __init__() missing 2 required positional arguments: 'platforms_dir' and 'cpe_items_dir' #9752

Review Hints:

• Alternate solution to Pass product 'platforms' and 'cpe_items' directory to ssg.template.Builder() when loading test scenarios in Automatus #9753
• Try to test a rule in a container or VM with and without this patch.
• Related:
  • https://github.com/ComplianceAsCode/content/actions/runs/3372700322/jobs/5596508651
  • https://github.com/ComplianceAsCode/content/actions/runs/3370997294/jobs/5592710001

[github-actions]

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

[mildas]

From CPE logic point of view, I can't provide my feedback.

From implementation point of view, platforms_dir and cpe_items_dir are being checked in constructor if they are defined - both parameters can be optional. With them being optional, None don't need to be passed toBuilder in RuleCheck.

[yuumasato]

From implementation point of view, platforms_dir and cpe_items_dir are being checked in constructor if they are defined - both parameters can be optional. With them being optional, None don't need to be passed toBuilder in RuleCheck.

I have made platforms_dir and cpe_items_dir optional.

[mildas]

Implementation looks good to me now.

[vojtapolasek]

I have a feeling that the correct way would be to somehow extract this functionality which locatest tests out of the builder class. In the end, Automatus is interested only in function get_all_tests. This function is not related to building content at all.
@mildas Do you think that the function relies so heavily on the Builder class that it has to stay there?

[mildas]

@mildas Do you think that the function relies so heavily on the Builder class that it has to stay there?

The function relies on the Builder class but it would be possible to move it somewhere else and setup the needed stuff there. But then we get to topics like where would it be more suitable or what about code duplication. That's out of scope of this fix.

[yuumasato]

@mildas @vojtapolasek Gimme some time, I'm working on a changeset to remove use of Builder from Automatus.

[vojtapolasek]

Then I think that I rather prefer this fix: #9753

[yuumasato]

@mildas @vojtapolasek I have relocated the test scenario functions away from Builder(), and removed its instantiation in Automatus.

[mildas]

Wrong variable?

$ python3 tests/test_suite.py rule --libvirt qemu:///session test-suite-rhel8 --datastream build/ssg-rhel8-ds.xml --no-report package_chrony_installed package_ntp_installed
WARNING - You call Automatus using the legacy 'test_suite.py' script, use the 'automatus.py' instead

Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/mlysonek/SCAP/content/logs/rule-custom-2022-11-03-1410/test_suite.log
WARNING - Nothing has been tested!
Traceback (most recent call last):
  File "/home/mlysonek/SCAP/content/tests/test_suite.py", line 21, in <module>
    ret = automatus.main()
  File "/home/mlysonek/SCAP/content/tests/automatus.py", line 498, in main
    options.func(options)
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/rule.py", line 665, in perform_rule_check
    checker.test_target()
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/oscap.py", line 699, in test_target
    self._test_target()
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/rule.py", line 445, in _test_target
    test_content_by_rule_id = self._get_test_content_by_rule_id(
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/rule.py", line 431, in _get_test_content_by_rule_id
    rule_test_content = self._get_rule_test_content(rule)
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/rule.py", line 413, in _get_rule_test_content
    all_tests = self._load_all_tests(rule)
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/rule.py", line 403, in _load_all_tests
    templated_tests = common.load_templated_tests(
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/common.py", line 498, in load_templated_tests
    test = load_test(path, template, local_env_yaml)
  File "/home/mlysonek/SCAP/content/tests/ssg_test_suite/common.py", line 508, in load_test
    template_parameters = templates[template_name].preprocess(
NameError: name 'templates' is not defined. Did you mean: 'template_vars'?

[yuumasato]

Nope, it is a global variable that I overlooked.
It is declared in ssg.templates and Builder() generates a dictionary during __init__().
https://github.com/ComplianceAsCode/content/blob/master/ssg/templates.py#L32

It looks like we cannot easily get rid of Builder(), :(

[yuumasato]

So Builder() goes through all of the templates in _SHARED_TEMPLATES and pre-loads all of them, this makes sense when building content.
For Automatus case, there is no need to load all of the templates whenever we are testing a templated rule. Aditionally, Builder() was being instantiated for each tested rule, so this probably improves performance a bit.

[yuumasato]

7d7fe67 loads the template and preprocesses it.

[codeclimate]

Code Climate has analyzed commit 7d7fe67 and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 46.7% (0.1% change).

View more on Code Climate.

[mildas]

Looks good to me. I've tested it with different combinations of Automatus and everything seems fine except #9762 - I was able to reproduce the issue also on 0.1.64 so it's not a regression and doesn't prevent us from merging this PR.