Update gradle test setup to consider test iterations for suite timeout

[dreamer-89]

Coming from #1561 (comment), existing test setup does not consider the test iteration count (-Dtests.iters) and uses 20 mins test suite timeout. This is problematic as it often leads to test suite timeout.

Alternative: Update timeout to a higher value. This is not an ideal solution.

[JalODan]

Hi, @dreamer-89. I tried to reproduce this issue, but could not. TimeoutSuite is set to 20 minute. Author of the issue says that these 20 minutes are shared by 100 iterations of the test. From this assumption it follows that, if execution time of 100 iterations exceeds 20 minutes a timeout would occur. But when I tried it locally, everything works fine:

From the image we can see that they run for longer than 20 minutes. It means, that each instance of the test running gets its own 20 minutes, and does not share it with other instances. And therefore, there is no such problem. Please correct me If am wrong.

[dreamer-89]

Hi, @dreamer-89. I tried to reproduce this issue, but could not. TimeoutSuite is set to 20 minute. Author of the issue says that these 20 minutes are shared by 100 iterations of the test. From this assumption it follows that, if execution time of 100 iterations exceeds 20 minutes a timeout would occur. But when I tried it locally, everything works fine: From the image we can see that they run for longer than 20 minutes. It means, that each instance of the test running gets its own 20 minutes, and does not share it with other instances. And therefore, there is no such problem. Please correct me If am wrong.

Thank you @JalODan for looking into this. This is interesting!

Good to know this is not a problem when tests are ran from IDE. Though, I see the issue was initially reported when test was ran with gradle command. Can you try running tests with gradle command below ?

./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards" -Dtests.seed=3ADB1F144D29EAD6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=America/Lower_Princes -Druntime.java=17 -Dtests.iters=100