Windows start up issues on main

[peternied]

Plugin install workflow seems to be seeing issues with Windows JDK 11/17. After several rerun attempts did not see any altered behavior and additional logging seems to indicate the security index isn't being created in a timely fashion.

Failure

From https://github.com/opensearch-project/security/actions/runs/3625326485/jobs/6113250804

Invoke-WebRequest: D:\a\_temp\2d6b1604-a1a3-410d-a119-3204283a8b44.ps1:6
Line |
   6 |  Invoke-WebRequest -SkipCertificateCheck -Uri 'https://localhost:9200/ …
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | OpenSearch Security not initialized.

Logs after node start

[2022-12-06T00:43:22,300][INFO ][o.o.s.OpenSearchSecurityPlugin] [fv-az206-877] Node started
[2022-12-06T00:43:22,304][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2022-12-06T00:43:22,304][INFO ][o.o.s.OpenSearchSecurityPlugin] [fv-az206-877] 0 OpenSearch Security modules loaded so far: []
[2022-12-06T00:43:22,304][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] Background init thread started. Install default config?: false
[2022-12-06T00:43:22,304][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] Try to load config ...
[2022-12-06T00:43:22,304][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] security index not exists (yet)
Error: 2-06T00:43:22,304][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,304][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,304][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Error: 2-06T00:43:22,320][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [fv-az206-877] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-12-06T00:43:32,341][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] Unable to load configuration due to java.util.concurrent.TimeoutException: Timeout after 10SECONDS while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT](index=.opendistro_security)
[2022-12-06T00:43:35,350][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] Try to load config ...
[2022-12-06T00:43:35,350][INFO ][o.o.s.c.ConfigurationRepository] [fv-az206-877] security index not exists (yet)
...

The Failure no such index lines repeat every 10 seconds as the background job attempts to bootstrap the cluster.

Impacted Pull Requests

• 6x failures Remove redundant test suite #2299
• 2x failures Automatically add untriaged label on issues #2297

[peternied]

@scrawfor99 Checkout #2299 as I beefed up some of the log output and captured the log data after the request was made to the cluster

[stephen-crawford]

@peternied I am looking at this now. I am not sure what changed that is causing the issue because it seemed to work fine before. I need to figure out how to make sure the index is populated since I am not sure how that works.

[cwperks]

There's an issue on core for this as well: opensearch-project/OpenSearch#5470

The fixes have been merged, this may be resolved as soon as another distribution build runs.

[stephen-crawford]

@cwperks thank you for letting me know about this. It looks like this may be a different issue sadly. There issue seems to only impact 1.3.7 whereas this in on main. We also never get the Error opening log file message they do. Maybe I am misunderstanding their issue though? What do you think?

If it is the same then that is great haha cause then we do not need to worry about this.