Re-organize tests/fmf-plans into a more concise format
#11809
Conversation
There are only a few test cases in the fedoraproject repo, all of them now abandoned due to the same logic being in Contest: Sanity/ansible-allowed-modules --> /static-checks/ansible/allowed-modules Sanity/smoke-test --> /static-checks/rpmbuild-ctest The two remaining fedoraproject tests (not run in Packit) are also covered by Contest: Sanity/ansible-machine-hardening --> /hardening/host-os/ansible Sanity/machine-hardening --> /hardening/host-os/oscap Signed-off-by: Jiri Jaburek <comps@nomail.dom>
All of these are running Contest anyway, might as well use a more conscise definition format, and put it into one file. Signed-off-by: Jiri Jaburek <comps@nomail.dom>
FMF has no concept of 'plans', that's a TMT feature. Signed-off-by: Jiri Jaburek <comps@nomail.dom>
This changes the TMT plan names from ie. /tests/tmt-plans/some/name to just /some/name Signed-off-by: Jiri Jaburek <comps@nomail.dom>
There are no remaining FMF/TMT tests that would run on Fedora. Signed-off-by: Jiri Jaburek <comps@nomail.dom>
Signed-off-by: Jiri Jaburek <comps@nomail.dom>
|
Start a new ephemeral environment with changes proposed in this pull request: Fedora Environment Oracle Linux 8 Environment |
|
The changes look fine and no problem on my side in regards of not running tests on Fedora anymore. However, the html-links tests seems quite flaky for GH CI, that's something to solve before merging this PR. I've also noticed it in productization that some links time out but are fine in re-run. And then there's the certificate problem which might be temporary issue but still it would block PRs. |
👍
Then we should probably just get rid of the test. Something that fails frequently enough that it would cause issues in pull requests, and is as insignificant as broken HTTP links in text, is not worth the combined days/weeks of human time investigating the failures. |
|
Are you fine with excluding the html-link test from here, then we can also exclude it from daily productization and run it only in stabilization test runs? |
Signed-off-by: Jiri Jaburek <comps@nomail.dom>
I've pushed a commit disabling it here, for now, to not block #11807 or whatever new PR I'll make with those changes. .. We can adjust the exclusion later if needed. |
|
🤖 A k8s content image for this PR is available at: Click here to see how to deploy itIf you alread have Compliance Operator deployed: Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and: |
|
Code Climate has analyzed commit 1bd04d6 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 59.2% (0.0% change). View more on Code Climate. |
|
As RHSecurityCompliance/contest#148 got merged, now we need to wait until #11796 is merged |
|
/packit build |
Description:
This is about cleaning up and re-organizing
tests/fmf-plans/into what I feel is a more concise and organized form.The end result should be nicely named TMT plans in Testing Farm (without
/tests/fmf-plansprefix), and with TMT HTML reports available in the Testing Farm data dir structureNote that this brings in a testing change - currently, some Beakerlib-based tests from fedoraproject.org run on Fedora via Testing Farm
Sanity/ansible-allowed-modulesSanity/smoke-test(rpmbuild + CTest)these would not be running on Fedora anymore, where Contest is not supported. However, given their nature, I believe running them on CentOS Stream is sufficient.
testing-farm:fedora-39-x86_64from the "required" Github checks.Review Hints: