[META] Cluster Manager Async Shard Fetch Revamp

[amkhar]

Describe the revamp

Original Issue opened/ Project goal : #5098

RFC/Proposal - #5098 (comment)
Please read above issue in detail before starting any discussion, thanks.

Impact of the change

We'll track following as subtasks of this goal. I'll keep adding new issues, if we find more ways to improve this behaviour.

• [Enhancement] Add support for async fetching of metadata for a batch of shards #8956
• [Enhancement] Create new transport action for fetching metadata of multiple primary & replica shards #8957
• [Enhancement] Enhance GatewayAllocator to handle a batch of shards fetching #8958
• [Enhancement] Enhance AllocationService to allocate unassigned shards in batches #8959
• [Enhancement] Add a batch allocator for building allocation decisions for multiple primary & replica shards #8960
• [Bug] Fix cluster health API response  #9020
• [Integ] Add integration test for Async Shard Fetch revamp #9021
• [Deprecate] Remove transport actions TransportNodesListGatewayStartedShards & TransportNodesListShardStoreMetadata #9022
• [Enhancement] Update TransportIndicesShardStoresAction to fetch the shard metadata using batch call #9023
• [Optimization] AsyncShardFetch response contains DiscoveryNode which takes GBs of memory #7861
• [Optimization] Reduce ShardId key size in AsyncShardFetch Revamp strategy #12010
• [Refactoring] Abstract AsyncShardFetch cache to allow restructuring for batch mode #12440
• [Restructure] Change Cache Structure in AsyncShardFetch class #12248

To Reproduce
Steps given in Draft PR #7269

Expected behavior
Cluster manager should be resilient to node restarts in high number.

[amkhar]

PRs and draft PRs raised for above sub tasks

1. Transport action request response class for primary Added transport action for bulk async shard fetch for primary shards #8218
2. Transport action request response class for replica Add batch async shard fetch transport action for replica #8218 #8356
3. PrimaryShardAllocator Refactoring to re-use existing code PrimaryShardAllocator refactor to abstract out shard state and method calls #9760
4. AsyncShardFetch class refactoring to re-use existing code Batch Async Fetcher class changes #8742
5. ReplicaShardAllocator Refactoring to re-use existing code Refactored the RSA to make it more extensible #10254
6. BaseGatewayShardsAllocator class - draft PR (BaseGatewayShardAllocator changes for Assigning the batch of shards #8776)
7. PrimaryShardBatchAllocator class - draft PR Add PrimaryShardBatchAllocator to take allocation decisions for a batch of shards #8916
8. ReplicaShardBatchAllocator class - draft PR Created new ReplicaShardBatchAllocator #8992
9. GatewayAllocator class - part 1 draft PR (Async Batch shards changes for GatewayAllocator #8746), part 2 draft PR (FetchData changes for primaries and replicas #8865)
10. AllocationService - actual calling is being done here. partial draft PR (Allocation service changes for batch assignment #8888)
11. Allocation Explain API - Fixed Allocation Explain API in batch mode #10348
12. Refactor AsyncShardFetch cache structure to allow batch mode cache restructuring Abstract AsyncShardFetch cache to allow restructuring for other caching strategies #12441
13. Add ShardBatchCache to handle batch transport action responses Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504