Feature: improve settings and environment logic and phase out redundant environment keys

[Donnype]

Changes

This improves the environment logic to pass all BOEFJES_* environment variables to the boefjes and allows you to override them with the KATalogus settings. This makes the flow more simple and intuitive.

Issue link

Closes #3356

Demo

QA notes

Please verify the Katalogus and boefjes such as dns-records still work properly, creating settings and review the documentation update.

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Looks good to me. Btw; this is very much related to #1906, any possibility we could discuss this one soon?

[ammar92]

Looks good to me.

For the next iteration, I suggest we start using pytest markers to distinguish between different test types (e.g., unit, integration, and performance tests). This approach will help us avoid using code/environment guards and streamline the CI process

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

Functionally this PR seems to work as expected. I enabled some general boefjes (onboarding + nmap ports) and they run. I also enabled LeaxIX with an API key and this runs as well, including the normalizers.

What doesn't work:

The docs have some room for improvement on how a user can work with the environment keys. Based on the initial ticket I would say you have to do the following:

Add BOEFJES_MYNEWBOEFJE=yourkey to the .env file, then in the schema.json file for MYNEWBOEFJE you would add the BOEFJES_MYNEWBOEFJE text and it should work? I think it would be very useful to have an example + description like this added, such that it is initially clear what/how it works.

Bug or feature?:

n/a

[Donnype]

@stephanie0x00 Done :)