Adding support for dynamically updating Leader/follower checker timeouts

[niyatiagg]

Description

Adding support for dynamically updating Leader/follower checker timeouts

Related Issues

Resolves #10377

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27472/
• CommitID: 6a2dbb9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29713/
• CommitID: 9a79455
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[niyatiagg]

@niyatiagg are you still working on this?

Yes, I am still on it. Have made the necessary changes and resolved conflicts!

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29714/
• CommitID: fbf2c36
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[owaiskazi19]

Looks like unrelated change. Let me rerun the check.

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.coordination.FollowersCheckerTests.testFailureCounterResetsOnSuccess" -Dtests.seed=9F1E3EE301F62571 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-DZ -Dtests.timezone=America/Inuvik -Druntime.java=21

org.opensearch.cluster.coordination.FollowersCheckerTests > testFailureCounterResetsOnSuccess FAILED
    java.lang.IllegalArgumentException: failed to parse value [84005ms] for setting [cluster.fault_detection.follower_check.timeout], must be <= [60000ms]
        at __randomizedtesting.SeedInfo.seed([9F1E3EE301F62571:ADD04656C38CBEB1]:0)
        at org.opensearch.common.settings.Setting.lambda$minMaxTimeValueParser$46(Setting.java:2721)
        at org.opensearch.common.settings.Setting$MinMaxTimeValueParser.apply(Setting.java:2652)
        at org.opensearch.common.settings.Setting$MinMaxTimeValueParser.apply(Setting.java:2606)
        at org.opensearch.common.settings.Setting.get(Setting.java:483)
        at org.opensearch.common.settings.Setting.get(Setting.java:477)
        at org.opensearch.cluster.coordination.FollowersChecker.<init>(FollowersChecker.java:146)
        at org.opensearch.cluster.coordination.FollowersCheckerTests.testBehaviourOfFailingNode(FollowersCheckerTests.java:395)
        at org.opensearch.cluster.coordination.FollowersCheckerTests.testFailureCounterResetsOnSuccess(FollowersCheckerTests.java:232)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29754/
• CommitID: fbf2c36
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[jainankitk]

@niyatiagg - It seems that changing from OpenSearchTestCase to SingleNodeTestCase has regressed unrelated tests. We already have initiative to reduce to flaky tests and don't want to increase their count. Hence, it might be better to revert to OpenSearchTestCase and add these new settings tests to separate class extending from SingleNodeTestCase.

[niyatiagg]

@niyatiagg - It seems that changing from OpenSearchTestCase to SingleNodeTestCase has regressed unrelated tests. We already have initiative to reduce to flaky tests and don't want to increase their count. Hence, it might be better to revert to OpenSearchTestCase and add these new settings tests to separate class extending from SingleNodeTestCase.

Makes sense. Will make the changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29770/
• CommitID: 672a371
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29806/
• CommitID: 8ca8da6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[niyatiagg]

@owaiskazi19 - The test failures seem unrelated to my changes. Can you please take a look?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/29824/
• CommitID: 8ca8da6

[jainankitk]

@niyatiagg - Can you also create tracking issue for updating the documentation? You can refer to one of the issue here: https://github.com/orgs/opensearch-project/projects/22

[niyatiagg]

@niyatiagg - Can you also create tracking issue for updating the documentation? You can refer to one of the issue here: https://github.com/orgs/opensearch-project/projects/22

Done!
opensearch-project/documentation-website#5603