Parametrization of rerunFailingTestsCount is not intuitive and user friendly

[rsvoboda]

Parametrization of rerunFailingTestsCount is not intuitive and user friendly, these are my concerns:

• Parametr is named just reruns - https://github.com/quarkus-qe/quarkus-test-framework/blob/main/pom.xml#L71
• By default I have 3 runs of the same test when it fails, make local investigation and development more cumbersome
• Reruns are something for CI and not for local test development or investigation
• Usability and OOTB experience should be the primary goal for us, especially when we want others to look into our stuff

There is already parametrization for surefire and failsafe available:

• https://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#rerunFailingTestsCount
• https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#rerunFailingTestsCount

I would rather see those params defined in CI jobs or in separate profile used in our CI.
The same applies to TS where we have 2 parameters ...

CC @michalvavrik @fedinskiy @mjurc

[michalvavrik]

Usability and OOTB experience should be the primary goal for us, especially when we want others to look into our stuff

if I create a reproducer, I simply add -Dreruns=0.

There is already parametrization for surefire and failsafe available:

it doesn't work for overriding. Both me and @fedinskiy tried it. I didn't try it for a scenario where reruns are not present in the POM.xml, but I am not in a favor of repetition in Jenkins jobs. Anyway I don't have a strong opinion, so let's go with whatever folk wants.

[fedinskiy]

I recently thought about it and I suggesting this:

1. Drop a separate parameter for openshift
2. Set reruns to zero in pom
3. Set reruns in Jenkins scripts to the desired number (eg 2 for OCP jobs and 3 for baremetal)

[rsvoboda]

I did a small experiment and passing failsafe.rerunFailingTestsCount works as expected

mvn -am -pl examples/restclient clean verify
mvn -am -pl examples/restclient clean verify -Dfailsafe.rerunFailingTestsCount=5

Code changes:

diff --git a/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java b/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
index d4178679..0c04b82a 100644
--- a/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
+++ b/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
@@ -17,6 +17,6 @@ public class PingPongResourceIT {

     @Test
     public void shouldPingPongWorks() {
-        ping.given().get("/ping/pong").then().statusCode(HttpStatus.SC_OK).and().body(is("pingpong"));
+        ping.given().get("/ping/pong").then().statusCode(HttpStatus.SC_OK).and().body(is("pingpongX"));
     }
 }
diff --git a/pom.xml b/pom.xml
index 4e95f52c..a2f09c8f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -68,7 +68,6 @@
         <infinispan-legacy.image>docker.io/infinispan/server:13.0</infinispan-legacy.image>
         <!-- TODO use official image if this fixed https://github.com/hashicorp/docker-consul/issues/184 -->
         <consul.image>docker.io/bitnami/consul:1.18.1</consul.image>
-        <reruns>2</reruns>
         <flaky-run-reporter.version>0.1.2.Beta1</flaky-run-reporter.version>
         <certificate-generator.version>0.6.0</certificate-generator.version>
     </properties>
@@ -233,7 +232,6 @@
                     <configuration>
                         <!-- fixes unavailable native binary build logs, https://github.com/quarkus-qe/quarkus-test-framework/issues/785 -->
                         <forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory" />
-                        <rerunFailingTestsCount>${reruns}</rerunFailingTestsCount>
                     </configuration>
                 </plugin>
                 <plugin>

[rsvoboda]

We can do this to make sure reruns are triggered on GitHub Actions CI

diff --git a/.github/mvn-settings.xml b/.github/mvn-settings.xml
index 32668345..daa5a051 100644
--- a/.github/mvn-settings.xml
+++ b/.github/mvn-settings.xml
@@ -4,6 +4,10 @@
     <profiles>
         <profile>
             <id>google-mirror-jboss-proxy</id>
+            <properties>
+                <failsafe.rerunFailingTestsCount>2</failsafe.rerunFailingTestsCount>
+                <surefire.rerunFailingTestsCount>2</surefire.rerunFailingTestsCount>
+            </properties>
             <repositories>
                 <repository>
                     <id>google-maven-central</id>

The same can be done for Jenkins based settings file, it would be global settings though.