Peer query speculative fixes and improvements #1430
Conversation
When using peerQuerySpeculative queries, if a request to a peer fails, instead of failing and aborting all requests, simply retry the request on another peer for the shard.
| @@ -546,6 +549,19 @@ func (s *Server) peerQuerySpeculativeChan(ctx context.Context, data cluster.Trac | |||
| } | |||
|
|
|||
| if resp.err != nil { | |||
| // check if there is another peer for this shardGroup. If so try it. | |||
There was a problem hiding this comment.
The reason I originally didn't retry errors, is that many errors aren't effective to retry. Things like bad requests will still fail with the peers. I would be interested to see how often this is actually helpful.
There was a problem hiding this comment.
Between when cluster.MembersForSpeculativeQuery() returned the list of peers and when the requests where actually run, nodes could go offline. We have been seeing bursts of failures in clusters that handle a high volume of requests because of this.
There was a problem hiding this comment.
no instance should issue a request that is considered a bad request by a peer. if we see that, it is a bug to be fixed.
There was a problem hiding this comment.
discussion on this topic lives in #985
fixup context handling for intra-cluster requests. The function for handling requests was using a depreciated approach for cancelling requests.
update peerQuerySpeculative to retry other peers in a shardGroup when a request fails.