[BUG] Calling Notifications from other plugins results in ClassLoader error

[joshuali925]

Describe the bug

common-utils/src/main/kotlin/org/opensearch/commons/notifications/NotificationsPluginInterface.kt

Line 199 in 4d8fb67

fun sendNotification(

Tested in reporting plugin. When reporting calls this function, the following error will show up.

java.lang.ClassCastException: class org.opensearch.commons.notifications.action.SendNotificationRequest cannot be cast to class org.opensearch.commons.notifications.action.SendNotificationRequest (org.opensearch.commons.notifications.action.SendNotificationRequest is in unnamed module of loader java.net.FactoryURLClassLoader @1229a2b7; org.opensearch.commons.notifications.action.SendNotificationRequest is in unnamed module of loader java.net.FactoryURLClassLoader @28f9fedd)
        at org.opensearch.notifications.action.SendNotificationAction.doExecute(SendNotificationAction.kt:47) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:136) [opensearch-index-management-1.0.0.0-rc1.jar:1.0.0.0-rc1]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
...

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[joshuali925]

https://discuss.elastic.co/t/how-to-share-classloader-bewteen-different-plugins/181076/ looks to be the same issue

[anirudha]

@dblock @dbbaughe @nknize please review, thanks

[dblock]

It looks like there's a cookbook for solving this exact issue in https://discuss.elastic.co/t/how-to-share-classloader-bewteen-different-plugins/181076/6 - have you tried it?

[dai-chen]

As the forum post above explains, this may be caused by "optimization to avoid serialization" for local request between plugins. Possible solutions include:

1. Copy common jar to OpenSearch lib at runtime and make each plugin compileOnly depends on it
2. Move common code to OpenSearch core engine codebase
3. Extend OpenSearch core engine to support this kind of communication
4. Force serialization/de-serialization request/response class even for local request (what's proposed in the post)

Will reproduce the issue and do a PoC for Option 4 first which seems not hacky and little impact on current architecture.