[BUG] java.io.OptionalDataException - Cache Timeout

[msoler8785]

What is the bug?
When the auth cache expires it isn't properly updated causing a java.io.OptionalDataException exception to be returned from OpenSearch.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Use the Bulk API after the plugins.security.cache.ttl_minutes period has elapsed.

What is the expected behavior?
The cache is refreshed and there are no errors.

What is your host/environment?

• OS: Ubuntu 20.04.4 LTS
• Version: 2.0.1
• Plugins: Standard

Do you have any additional context?

• Please see: [BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc #1961
• Issue can cause data loss if using Logstash.

[msoler8785]

I suspect this PR to be the culprit, but I don't know enough to pinpoint the problem: ef0d40a

[peternied]

[Triage] Thanks for filing this issue @msoler8785 we will look into this issue.

[TheMeier]

I suspect this might be the reason for a problem we are seeing. We have installed a fresh cluster version 2.1.0. We deliver logs with fluent-bit. Every hour the delivery fails with:

"status":500,"error":{"type":"exception","reason":"java.io.OptionalDataException","caused_by":{"type":"i_o_exception","reason":null}}}}

It fails for roughly one hour and starts working again. Then one hour later the cycle repeats.
Clearing the cache (via dashboards or curl) does not resolve the problem.

What does resolve the problem ist to run securityadmin.sh with the exact same permissions again.

[TheMeier]

What does resolve the problem ist to run securityadmin.sh with the exact same permissions again.

Even that does not reliably fix the problem.
Also I raised plugins.security.cache.ttl_minutes. Still seeing this problem ~10 times per day

[cwperks]

Closing as #1970 has been merged