feat(ingester): restrict /flush and /shutdown endpoints to POST only by Goutham-Annem · Pull Request #7674 · cortexproject/cortex

Goutham-Annem · 2026-07-05T17:08:11Z

What this PR does:

The /ingester/flush, /ingester/shutdown, /flush, and /shutdown endpoints previously accepted GET requests, making it possible to accidentally trigger a destructive ingester operation by clicking a link in a browser or via a pre-fetch proxy following the index page's links.

This PR makes three coordinated changes:

Route restriction (pkg/api/api.go): The four flush/shutdown RegisterRoute calls now only list "POST". A browser GET (or curl without -X POST) returns 405 Method Not Allowed.
Index page template (pkg/api/handlers.go): Links in the Dangerous: section now render as <form method="POST"><button> elements instead of <a href> anchors. This ensures that clicking in the browser issues a POST, not a GET. Non-dangerous links continue to render as plain anchors. A template function IsDangerous(section string) bool drives the branching.
Migration script (tools/migrate-ingester-statefulsets.sh): The BusyBox wget invocation inside kubectl exec is updated to wget --post-data="" -T 5 .... BusyBox wget supports --post-data, so the script continues to work without requiring curl inside the Cortex container image (which is unavailable per the original issue discussion).

Which issue(s) this PR fixes:
Fixes #3243

Checklist

Tests updated — TestIndexPageContent extended with assertions that dangerous links produce method="POST" forms and no <a href>; new TestIndexHandlerDangerousLinksUsePostForms covers prefix behaviour for dangerous links
CHANGELOG.md updated
Documentation added (N/A — behaviour is visible in the UI itself)

Note: This PR was developed with AI assistance (Claude Code).

friedrichg · 2026-07-08T21:33:10Z

this should trigger a change in the docs too https://cortexmetrics.io/docs/api/#shutdown

Goutham-Annem · 2026-07-18T06:43:37Z

Gentle ping — CI is passing, happy to address any feedback.

Goutham-Annem · 2026-07-18T17:34:01Z

Gentle ping — CI is passing and there are no conflicts. Happy to address any feedback!

friedrichg

I am ok with this.
But also brings the question of what should happen with /ingester/mode and /ingester/renewTokens

Goutham-Annem · 2026-07-22T07:16:57Z

Good point — extended the PR to also restrict /ingester/renewTokens and /ingester/mode to POST-only. Both are already in SectionDangerous, so the index page template already renders them as POST forms (including the query string in the action attribute, so mode=READONLY / mode=ACTIVE still work). No test changes needed since neither endpoint has dedicated API tests.

SungJin1212 · 2026-07-23T06:38:42Z

@Goutham-Annem
lgtm, can you share a new UI?

friedrichg

One minor nit. Thanks!

friedrichg · 2026-07-23T22:25:27Z

 # Changelog

 ## master / unreleased
+* [CHANGE] Ingester: `/ingester/flush`, `/ingester/shutdown`, `/flush`, and `/shutdown` endpoints now only accept `POST` requests. The index page renders these as HTML form buttons to prevent accidental browser-triggered GET requests. Update any scripts using `wget` or `curl` without `-X POST` to use `curl -X POST` or `wget --post-data=""`. #3243


we should probably also mention /ingester/renewTokens and /ingester/mode

Kindly adjust the changelog to include this @Goutham-Annem

@Goutham-Annem
Can you update the changelog? (Adds /ingester/renewTokens, /ingester/mode, and PR number (#7674))

Goutham-Annem · 2026-07-24T05:48:53Z

Friendly ping — this PR was approved by @friedrichg and is MERGEABLE with CI passing. Could a maintainer merge it when ready? Thank you!

The /ingester/flush, /ingester/shutdown, /flush, and /shutdown endpoints previously accepted GET requests, making it possible to accidentally trigger a destructive ingester operation by clicking a link in a browser or by a pre-fetch proxy following links on the index page. Changes: - Route registration for flush/shutdown changed from ["GET","POST"] to ["POST"]. A browser GET (or curl without -X POST) now returns 405 Method Not Allowed. - Index page template updated to render links in the "Dangerous:" section as HTML <form method="POST"> submit buttons rather than plain <a href> anchors, preventing browser-initiated GET requests while keeping the UI functional. - tools/migrate-ingester-statefulsets.sh updated to use `wget --post-data=""` (BusyBox wget supports this flag) so the script continues to work after the GET handler is removed. - Two tests added/extended to assert the POST-form rendering for dangerous links. Fixes cortexproject#3243 Signed-off-by: Goutham Annem <gouthemannem@gmail.com>

Reflects the change introduced in pkg/api/api.go that removed GET from the allowed methods for /ingester/flush, /flush, /ingester/shutdown, and /shutdown. Browsers and scripts must now use POST to trigger these operations. Signed-off-by: Goutham Annem <gouthemannem@gmail.com>

The /flush endpoint now only accepts POST (changed in feat cortexproject#3243). Update the integration test to call e2e.PostRequest instead of e2e.GetRequest so the test stays consistent with the enforced method restriction. Signed-off-by: Goutham Annem <gouthemannem@gmail.com>

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>

sandy2008

The route and index-page changes look right to me, and the TestIndex* tests pass locally. Before this goes in, though, I think the Kubernetes side needs a migration path. Details inline.

sandy2008 · 2026-10-28T11:50:46Z

-	a.RegisterRoute("/ingester/shutdown", http.HandlerFunc(i.ShutdownHandler), false, "GET", "POST")
-	a.RegisterRoute("/ingester/renewTokens", http.HandlerFunc(i.RenewTokenHandler), false, "GET", "POST")
+	a.RegisterRoute("/ingester/flush", http.HandlerFunc(i.FlushHandler), false, "POST")
+	a.RegisterRoute("/ingester/shutdown", http.HandlerFunc(i.ShutdownHandler), false, "POST")


This breaks the ingester preStop hook from our getting-started Helm values (docs/getting-started/cortex-values.yaml: lifecycle.preStop.httpGet.path: "/ingester/shutdown"), and the cortex-helm-chart ships the same default. Kubernetes httpGet hooks always send GET, so after this change the hook gets a 405 and the pod is terminated without the flush and ring unregistration that /shutdown does. Since #7637 the image is distroless, so an exec hook can't POST either. Could the PR update that values file, e.g. drop the hook and set -blocks-storage.tsdb.flush-blocks-on-shutdown=true (-ingester.unregister-on-shutdown already defaults to true)? It would also help to call this out in the CHANGELOG entry so chart users change theirs before upgrading. That entry still needs the /ingester/mode, /ingester/renewTokens and #7674 update requested above.

sandy2008 · 2026-10-28T11:50:46Z

  # -T causes wget to wait only 5 seconds, otherwise /shutdown takes a long time.
  # Preferably we would wait for /shutdown to return, but unfortunately that doesn't work (even with big timeout), wget complains with weird error.
-  kubectl exec "$POD_TO_SHUTDOWN" --namespace="$NAMESPACE" -- wget -T 5 http://localhost:80/shutdown >/dev/null 2>/dev/null || true
+  kubectl exec "$POD_TO_SHUTDOWN" --namespace="$NAMESPACE" -- wget --post-data="" -T 5 http://localhost:80/shutdown >/dev/null 2>/dev/null || true


Heads-up: since #7637 the Cortex image is gcr.io/distroless/static-debian12 (cmd/cortex/Dockerfile), so there's no wget or shell in the container. This kubectl exec can't run against 1.22+ images, with or without --post-data. It's not something this PR introduced, but the script won't keep working the way the description says. Going through kubectl port-forward and calling curl -X POST from outside might be the way to fix it.

sandy2008 · 2026-10-28T11:50:46Z

+| [Shutdown](#shutdown) | Ingester || `POST /ingester/shutdown` |
 | [Ingesters ring status](#ingesters-ring-status) | Ingester || `GET /ingester/ring` |
 | [Ingester tenants stats](#ingester-tenants-stats) | Ingester || `GET /ingester/all_user_stats` |
 | [Ingester mode](#ingester-mode) | Ingester || `GET,POST /ingester/mode` |


/ingester/mode is POST-only now as well (pkg/api/api.go:340), but this row and the ### Ingester mode section further down still say GET,POST /ingester/mode.

pull-request-size Bot added the size/M label Jul 5, 2026

dosubot Bot added component/ingester type/security labels Jul 5, 2026

friedrichg approved these changes Jul 21, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 21, 2026

friedrichg requested a review from danielblando July 21, 2026 23:36

SungJin1212 approved these changes Jul 23, 2026

View reviewed changes

friedrichg approved these changes Jul 23, 2026

View reviewed changes

Goutham-Annem added 4 commits September 10, 2026 14:17

fix: restrict /ingester/renewTokens and /ingester/mode to POST only

47d6ae4

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>

Goutham-Annem force-pushed the fix/3243-post-only-flush-shutdown branch from ac92f88 to 47d6ae4 Compare September 10, 2026 21:17

Goutham-Annem requested a review from a team as a code owner September 10, 2026 21:17

sandy2008 reviewed Oct 28, 2026

View reviewed changes

Conversation

Goutham-Annem commented Jul 5, 2026

Uh oh!

friedrichg commented Jul 8, 2026

Uh oh!

Goutham-Annem commented Jul 18, 2026

Uh oh!

Goutham-Annem commented Jul 18, 2026

Uh oh!

friedrichg left a comment

Choose a reason for hiding this comment

Uh oh!

Goutham-Annem commented Jul 22, 2026

Uh oh!

SungJin1212 commented Jul 23, 2026

Uh oh!

friedrichg left a comment

Choose a reason for hiding this comment

Uh oh!

friedrichg Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

friedrichg Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

SungJin1212 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

Goutham-Annem commented Jul 24, 2026

Uh oh!

sandy2008 left a comment

Choose a reason for hiding this comment

Uh oh!

sandy2008 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

sandy2008 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

sandy2008 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants