Skip to content

fix(querier): de-flake store-gateway limits integration tests by waiting for ACTIVE store-gateway in querier ring view #7614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sandy2008 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix(querier): de-flake store-gateway limits integration tests by waiting for ACTIVE store-gateway in querier ring view #7614

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
* [BUGFIX] Query Frontend: Fix native histogram responses not being handled correctly in `minTime()` sort ordering for split_by_interval merge. #7555
* [BUGFIX] Distributor: Release the push worker pool goroutines on shutdown by stopping the async executor during the stopping phase when `-distributor.num-push-workers` is set. #7602
* [BUGFIX] Querier: Fix unbounded resource leak in the bucket-scan blocks finder (used when the bucket index is disabled). Per-tenant metadata fetchers, their Prometheus registries, and on-disk meta caches are now evicted once a tenant is no longer active, instead of being retained for the lifetime of the process. #7573
* [BUGFIX] Querier: Fix flake in integration tests TestQuerierWithStoreGatewayDataBytesLimits and TestQuerierWithBlocksStorageLimits by waiting for the querier to see the store-gateway ACTIVE in the ring before querying. #7614

## 1.21.0 2026-04-24

Expand Down
16 changes: 16 additions & 0 deletions integration/querier_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -474,6 +474,14 @@ func TestQuerierWithBlocksStorageLimits(t *testing.T) {
require.NoError(t, storeGateway.WaitSumMetrics(e2e.Equals(512), "cortex_ring_tokens_total"))
require.NoError(t, storeGateway.WaitSumMetrics(e2e.Equals(1), "cortex_bucket_store_blocks_loaded"))

// Wait until the store-gateway is ACTIVE in the querier's view of the store-gateway ring. The
// store-gateway registers JOINING (with tokens) and switches to ACTIVE only after the initial
// blocks sync, so the waits above can pass while the querier would still fail queries with
// "at least 1 healthy replica required, could only find 0" (HTTP 500) instead of the expected 422.
require.NoError(t, querier.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
labels.MustNewMatcher(labels.MatchEqual, "name", "store-gateway-client"),
labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))

// Query back the series.
c, err = e2ecortex.NewClient("", querier.HTTPEndpoint(), "", "", "user-1")
require.NoError(t, err)
Expand Down Expand Up @@ -571,6 +579,14 @@ func TestQuerierWithStoreGatewayDataBytesLimits(t *testing.T) {
require.NoError(t, storeGateway.WaitSumMetrics(e2e.Equals(512), "cortex_ring_tokens_total"))
require.NoError(t, storeGateway.WaitSumMetrics(e2e.Equals(1), "cortex_bucket_store_blocks_loaded"))

// Wait until the store-gateway is ACTIVE in the querier's view of the store-gateway ring. The
// store-gateway registers JOINING (with tokens) and switches to ACTIVE only after the initial
// blocks sync, so the waits above can pass while the querier would still fail queries with
// "at least 1 healthy replica required, could only find 0" (HTTP 500) instead of the expected 422.
require.NoError(t, querier.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_ring_members"}, e2e.WithLabelMatchers(
labels.MustNewMatcher(labels.MatchEqual, "name", "store-gateway-client"),
labels.MustNewMatcher(labels.MatchEqual, "state", "ACTIVE"))))

// Query back the series.
c, err = e2ecortex.NewClient("", querier.HTTPEndpoint(), "", "", "user-1")
require.NoError(t, err)
Expand Down
Loading