[sandy2008]

What this PR does

Fixes #7572.

BucketScanBlocksFinder (the querier's block discovery used when the bucket index is disabled) lazily creates one block.MetadataFetcher per tenant and caches it in d.fetchers. Each fetcher owns a per-tenant prometheus.Registry (registered into the aggregating MetadataFetcherMetrics via AddUserRegistry) and an on-disk meta cache under <sync-dir>/<userID>/meta-syncer/. These entries were never removed, so over a process's lifetime the fetcher map, the per-tenant Prometheus registries, and the on-disk meta caches grew unbounded as tenants were deleted from storage — even though scanBucket already prunes the block-metadata maps for deleted tenants.

Fix

After each scan, scanBucket reconciles d.fetchers against the active-user set returned by ScanUsers:

• evict the cached fetcher and delete() it from the map;
• unregister its per-tenant Prometheus registry via MetadataFetcherMetrics.RemoveUserRegistry (this already existed — the querier just never called it);
• delete the fetcher's on-disk meta-syncer cache.

Design notes (surfaced and hardened during review):

• Decoupled from per-tenant scan errors. The active set comes from a successful ScanUsers (the scan returns early if that fails), so a tenant being evicted is by construction absent from resErrs. Eviction therefore runs on every scan where the context isn't cancelled — not only on fully-successful scans — keeping the leak bounded even under churn with transient per-tenant errors.
• Single-binary safety. In single-binary mode CacheDir is the store-gateway's SyncDir, whose block data lives at <sync-dir>/<userID>/<blockID>/. The cleanup therefore removes only the fetcher's own <userID>/meta-syncer sub-directory (plus a best-effort os.Remove of the now-empty parent), never the per-tenant tree, provably preserving co-located block data.
• Mirrors the store-gateway's existing per-tenant lifecycle (bucket_stores.go: RemoveUserRegistry + deleteLocalFilesForExcludedTenants). The bucket-index blocks finder is unaffected — it already idle-prunes its per-tenant loaders.

Tests

Added unit tests covering fetcher-map eviction, per-tenant Prometheus registry removal (the cortex_blocks_meta_synced gauge series drop to zero), on-disk meta-syncer removal, preservation of co-located store-gateway block data under the same per-tenant directory, multi-tenant selectivity + re-creation of a returning tenant, and eviction continuing to run when another tenant's scan errors. All pass under -race.

Out of scope (potential follow-ups)

• On the chronic partial-error path, userMetas/userMetasLookup/userDeletionMarks are merged (maps.Copy) rather than pruned, so block metadata for tenants that disappear while some other tenant keeps failing to scan is retained until the next fully-successful scan. This is pre-existing and is not a registry/heap leak.
• On-disk meta caches orphaned by tenants deleted while a querier-only deployment was down are not reaped by the new process (in single-binary mode the store-gateway reaps them). Minor and disk-only.

Checklist

• CHANGELOG.md updated ([BUGFIX] Querier)
• Tests added; pass under -race
• gofmt/go vet clean
• make doc — not needed (no flag/config changes)
• Commit signed off (DCO)

AI assistance disclosure

Per the Generative AI Contribution Policy: this change was developed with AI assistance (Claude Code), including a multi-round review pass. I have reviewed and validated every line, the root-cause analysis, and the tests, and take full responsibility for the contribution.

🤖 Generated with Claude Code

[friedrichg]

Thanks!.

Bucket index disabled is not recommended for production. it's very bad in terms of performance.But I am ok fixing bugs in it.

[SungJin1212]

lgtm