[BUG] org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky #12664

reta · 2024-03-14T12:58:51Z

Describe the bug

A number of org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky:

java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null
	at __randomizedtesting.SeedInfo.seed([65E3DFFD00C5A81B:1710CEB101EDB3A6]:0)
	at org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionEnforcedMode(AdmissionControlSingleNodeTests.java:102)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Other

To Reproduce

./gradlew ':server:test' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionEnforcedMode" -Dtests.seed=65E3DFFD00C5A81B

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/34896/

The text was updated successfully, but these errors were encountered:

reta · 2024-03-14T13:00:08Z

@ajaymovva please take a look, caused by #12583

ajaymovva · 2024-03-14T14:02:31Z

Not able to reproduce in 300 iterations debugging further

➜  OpenSearch git:(main) ./gradlew ':server:test' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionMonitorOnlyMode" -Dtests.seed=14E018DE8C6ECC4B -Dtests.iters=300

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.6
  OS Info               : Mac OS X 14.3.1 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/movvaam/.sdkman/candidates/java/21.0.2-amzn
  Random Testing Seed   : 14E018DE8C6ECC4B
  In FIPS 140 mode      : false
=======================================

> Task :server:test
WARNING: Using incubator modules: jdk.incubator.vector
Mar 14, 2024 7:21:10 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/Users/movvaam/AWS/OpenSource/core/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/Users/movvaam/.gradle/wrapper/dists/gradle-8.6-all/3mbtmo166bl6vumsh5k2lkq5h/gradle-8.6/lib/plugins/gradle-testing-base-8.6.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 8m 35s

reta · 2024-03-14T14:13:39Z

Not able to reproduce in 300 iterations debugging further

Please try to reproduce without -Dtests.seed=14E018DE8C6ECC4B , the issue is there

ajaymovva · 2024-03-14T14:22:19Z

On Mac:

➜  OpenSearch git:(main) ./gradlew ':server:test' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionEnforcedMode" -Dtests.iters=100                              

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.6
  OS Info               : Mac OS X 14.3.1 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/movvaam/.sdkman/candidates/java/21.0.2-amzn
  Random Testing Seed   : 732CED713A830FB3
  In FIPS 140 mode      : false
=======================================

> Task :server:test
WARNING: Using incubator modules: jdk.incubator.vector
Mar 14, 2024 7:44:48 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>

BUILD SUCCESSFUL in 2m 37s
54 actionable tasks: 1 executed, 53 up-to-date

➜  OpenSearch git:(main) ./gradlew ':server:test' --tests "org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionMonitorOnlyMode" -Dtests.iters=100

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.6
  OS Info               : Mac OS X 14.3.1 (aarch64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/movvaam/.sdkman/candidates/java/21.0.2-amzn
  Random Testing Seed   : B23CAC66FF9FC526
  In FIPS 140 mode      : false
=======================================

> Task :server:test
WARNING: Using incubator modules: jdk.incubator.vector
Mar 14, 2024 7:48:36 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>

BUILD SUCCESSFUL in 2m 37s
54 actionable tasks: 1 executed, 53 up-to-date

reta · 2024-03-14T14:23:25Z

@ajaymovva this is a problem with a flaky tests - it is difficult to reproduce, another occurrence:

https://build.ci.opensearch.org/job/gradle-check/34905/testReport/junit/org.opensearch.ratelimitting.admissioncontrol/AdmissionControlSingleNodeTests/testAdmissionControlRejectionEnforcedMode/

…amework (opensearch-project#12583)" This reverts commit b6b16d8. Reverting as this introduced test failures detailed in opensearch-project#12664.

andrross · 2024-03-14T18:02:48Z

I created a revert PR #12670 to unblock things while we figure out the test failures

…amework (opensearch-project#12583)" This reverts commit b6b16d8. Reverting as this introduced test failures detailed in opensearch-project#12664. Signed-off-by: Andrew Ross <andrross@amazon.com>

…amework (opensearch-project#12583) (opensearch-project#12659)" This reverts commit 3970ec4. Reverting as this introduced test failures detailed in opensearch-project#12664. Signed-off-by: Andrew Ross <andrross@amazon.com>

…amework (#12583)" (#12670) This reverts commit b6b16d8. Reverting as this introduced test failures detailed in #12664. Signed-off-by: Andrew Ross <andrross@amazon.com>

…amework (#12583) (#12659)" (#12671) This reverts commit 3970ec4. Reverting as this introduced test failures detailed in #12664. Signed-off-by: Andrew Ross <andrross@amazon.com>

reta · 2024-03-14T19:33:45Z

Closing, the change has been reverted f37b6ac

harishbhakuni · 2024-03-14T22:55:26Z

test org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.testAdmissionControlRejectionMonitorOnlyMode is still flaky and failing here: https://build.ci.opensearch.org/job/gradle-check/34952/

reta · 2024-03-14T23:14:24Z

@harishbhakuni please rebase against the target branch, the commits which introduced the issue where reverted

…amework (opensearch-project#12583)" (opensearch-project#12670) This reverts commit b6b16d8. Reverting as this introduced test failures detailed in opensearch-project#12664. Signed-off-by: Andrew Ross <andrross@amazon.com>

…amework (opensearch-project#12583)" (opensearch-project#12670) This reverts commit b6b16d8. Reverting as this introduced test failures detailed in opensearch-project#12664. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Mar 14, 2024

github-actions bot added the Other label Mar 14, 2024

reta assigned ajaymovva Mar 14, 2024

reta mentioned this issue Mar 14, 2024

Bump Jackson version from 2.16.2 to 2.17.0 #12662

Merged

8 tasks

amkhar mentioned this issue Mar 14, 2024

BaseGatewayShardAllocator changes for Assigning the batch of shards #8776

Merged

6 tasks

This was referenced Mar 14, 2024

[FEATURE] Built-in secure transports support #12435

Merged

[Backport] [2.x] Bump Jackson version from 2.16.2 to 2.17.0 (#12662) #12668

Merged

[AUTOCUT] Gradle Check Failure on push to main #12669

Closed

andrross mentioned this issue Mar 14, 2024

Revert "Integrate IO Based AdmissionController to AdmissionControl Fr… #12670

Merged

8 tasks

andrross mentioned this issue Mar 14, 2024

[2.x] Revert "Integrate IO Based AdmissionController to AdmissionControl Fr… #12671

Merged

8 tasks

reta closed this as completed Mar 14, 2024

harishbhakuni reopened this Mar 14, 2024

harishbhakuni mentioned this issue Mar 14, 2024

Simplify remote directory cleanup after snapshot delete to … #12672

Merged

8 tasks

reta closed this as completed Mar 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky #12664

[BUG] org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky #12664

reta commented Mar 14, 2024

reta commented Mar 14, 2024

ajaymovva commented Mar 14, 2024

reta commented Mar 14, 2024

ajaymovva commented Mar 14, 2024

reta commented Mar 14, 2024

andrross commented Mar 14, 2024

reta commented Mar 14, 2024

harishbhakuni commented Mar 14, 2024

reta commented Mar 14, 2024

[BUG] org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky #12664

[BUG] org.opensearch.ratelimitting.admissioncontrol.AdmissionControlSingleNodeTests.* are flaky #12664

Comments

reta commented Mar 14, 2024

Describe the bug

Related component

To Reproduce

Expected behavior

Additional Details

reta commented Mar 14, 2024

ajaymovva commented Mar 14, 2024

reta commented Mar 14, 2024

ajaymovva commented Mar 14, 2024

reta commented Mar 14, 2024

andrross commented Mar 14, 2024

reta commented Mar 14, 2024

harishbhakuni commented Mar 14, 2024

reta commented Mar 14, 2024