[BUG] [Remote State] Cluster chaining can break if there is consecutive node replace and restart

[linuxpi]

Describe the bug
Cluster chaining can break if there is consecutive node replace and restart

To Reproduce
Step for single node cluster

1. Bootstrap single node

2. No local disk state found

3. No remote disk state found

4. Cluster Manager Election success

5. First cluster uuid formed - uuid_1

6. First pre-publish - write Metadata to remote

7. First publish - write Metadata to disk with clusterUUID uuid_1 and clusterUUIDCommitted as false

8. First pre-commit - write Metadata to remote with clusterUUID uuid_1 and clusterUUIDCommitted as true
   first commit - write Metadata to local disk with clusterUUID commmitted true

9. Terminate the node and replace with a new one so that all local disk state is lost

10. Bootstrap new replacement node

11. No local disk state found, hence no cluster uuid.

12. Go to remote for latest state.

13. Found Metadata in remote with clusterUUID uuid_1 and clusterUUIDCommitted as true

14. Construct ClusterState from remote Metadata with clusterUUID ClusterState.UNKNOWN_UUID and clusterUUIDCommitted as false.

15. Write new ClusterState to disk with clusterUUID ClusterState.UNKNOWN_UUID and clusterUUIDCommitted as false.

16. Cluster Manager Election success

17. New cluster uuid formed - uuid_2

18. Pre publish - write to remote with clusterUUID uuid_2 and clusterUUIDCommitted as false.

19. Publish - write to local disk with clusterUUID uuid_2 and clusterUUIDCommitted as false.

20. Kill process on the node.

21. Process restarts and local disk state is found with clusterUUID uuid_2 and clusterUUIDCommitted as false.

22. Don't go to remote as clusterUUID is not ClusterState.UNKNOWN_UUID .

23. lastKnownClusterUUID remains ClusterState.UNKNOWN_UUID and RemotePersistedState is initialized with previousClusterUUID as ClusterState.UNKNOWN_UUID .

24. First pre-commit after this will write state to remote by calling RemotePersistedState.setLastAcceptedState . This will write new ClusterState to remote with previousClusterUUID as ClusterState.UNKNOWN_UUID

OpenSearch/server/src/main/java/org/opensearch/gateway/GatewayMetaState.java

Lines 165 to 189 in 9b7a9d0

	if (DiscoveryNode.isClusterManagerNode(settings) && isRemoteStoreClusterStateEnabled(settings)) {
	// If the cluster UUID loaded from local is unknown (_na_) then fetch the best state from remote
	// If there is no valid state on remote, continue with initial empty state
	// If there is a valid state, then restore index metadata using this state
	String lastKnownClusterUUID = ClusterState.UNKNOWN_UUID;
	if (ClusterState.UNKNOWN_UUID.equals(clusterState.metadata().clusterUUID())) {
	lastKnownClusterUUID = remoteClusterStateService.getLastKnownUUIDFromRemote(
	clusterState.getClusterName().value()
	);
	if (ClusterState.UNKNOWN_UUID.equals(lastKnownClusterUUID) == false) {
	// Load state from remote
	final RemoteRestoreResult remoteRestoreResult = remoteStoreRestoreService.restore(
	// Remote Metadata should always override local disk Metadata
	// if local disk Metadata's cluster uuid is UNKNOWN_UUID
	ClusterState.builder(clusterState).metadata(Metadata.EMPTY_METADATA).build(),
	lastKnownClusterUUID,
	false,
	new String[] {}
	);
	clusterState = remoteRestoreResult.getClusterState();
	}
	}
	remotePersistedState = new RemotePersistedState(remoteClusterStateService, lastKnownClusterUUID);
	}

Expected behavior
previousClusterUUID chaining should not break in any case.