Cache vertical shards in query frontend #5648
Conversation
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| // Allow other requests to execute | ||
| lock.Unlock() | ||
| <-time.After(200 * time.Millisecond) |
There was a problem hiding this comment.
The query frontend fanout seems to cancel all requests as soon as one request fails. Because of that, it is hard to provoke partial failures since the first failure will cause all parallel requests to fail.
This like adds a delay to failures so that successful requests can succeed, but it can still lead to flakes. Not sure if there is a better option to test this scenario.
There was a problem hiding this comment.
Mhm, maybe we could check if numFailures == 0 { waitForAnotherRequestToCome(); } somehow? Perhaps a shared sync.Cond could be used?
There was a problem hiding this comment.
This is a neat trick, seems to be a much better solution 👍
There was a problem hiding this comment.
Hm, looks like it's not possible to add this coordination here because we need to release other threads when the sharding middleware has seen a successful response. If we release them here, failed requests can still complete before successful ones.
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
f25e569
to
445e2d0
Compare
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| testutil.Equals(t, tc.expected, *res) | ||
|
|
||
| //if *res > tc.expected { |
pkg/queryfrontend/roundtrip_test.go
Outdated
|
|
||
| // Allow other requests to execute | ||
| lock.Unlock() | ||
| <-time.After(200 * time.Millisecond) |
There was a problem hiding this comment.
Mhm, maybe we could check if numFailures == 0 { waitForAnotherRequestToCome(); } somehow? Perhaps a shared sync.Cond could be used?
The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
348a8b3
to
cd96510
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
fpetkovski
force-pushed
the
vertical-sharding-caching
branch
from
cd96510
to
9962948
Compare
* Cache vertical shards in query frontend The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Adjust cache key tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove source of flakiness by using sync.Cond Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Prakul Jain <prakul.jain@udaan.com>
* Cache vertical shards in query frontend The vertical sharding middleware is currently executed after the caching middleware. Because of this, individual vertical shards are not getting cached when the succeed. Caching is only done when the entire requests including all shards complete successfully. This commit moves the vertical sharding middleware before the caching middleware. It also modifies caching keys to contain the total shards and the shard number so that each vertical shard gets an independent caching key. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Adjust cache key tests Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Remove source of flakiness by using sync.Cond Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Prakul Jain <prakul.jain@udaan.com>
The vertical sharding middleware is currently executed after the
caching middleware. Because of this, individual vertical shards are
not getting cached when the succeed. Caching is only done when the
entire requests including all shards complete successfully.
This commit moves the vertical sharding middleware before the caching
middleware. It also modifies caching keys to contain the total shards
and the shard number so that each vertical shard gets an independent
caching key.
Changes
Enable caching in query frontend for individual vertical shards.
Verification
Added unit tests and tests locally by purposefully causing failures in queriers.