Skip to content

Derive the backend-svc peer cap from the backend tier - #10

Open
zhanghaohit wants to merge 1 commit into
mainfrom
fix/backend-svc-peer-cap
Open

zhanghaohit wants to merge 1 commit into
mainfrom
fix/backend-svc-peer-cap

Conversation

@zhanghaohit

Copy link
Copy Markdown
Collaborator

Problem

The backend-svc tier is a single Service VIP in front of the same backends as the pod-IP backend tier. It was rendered without a per-peer maximum unless the ModelRoute set maxConcurrency on it, and the sglang / vllm charts don't. openresty then used its default_max (20) for that peer. A rendered route looked like this:

{ "10.105.43.217",  8000,  "cart-0",        3, 200, probe = "/health" },  -- 100 x 2 backends
{ "192.168.3.113",  30000, "a100-5",        2, 100, ... },
{ "192.168.7.247",  30000, "h800-39",       2, 100, ... },
{ "10.101.113.190", 30000, "backend-svc-0", 1 },                          -- no cap -> default_max 20

Normally that tier carries nothing. But when every cart and pod-IP peer is banned and traffic falls back to the VIP, the whole route is capped at 20 in-flight requests, while the backends behind it can take maxConcurrency x N.

Change

  • Derived cap: a backend-svc peer without its own maxConcurrency gets the backend tier's maxConcurrency x number of backends, the same rule a cart peer with maxConcurrencyFromBackend already follows. It tracks scaling too.
  • Explicit value wins: an explicit maxConcurrency on backend-svc is used as-is.
  • No backend cap: with no backend cap, or no backends, the value stays 0 and openresty's default_max applies, as before.

The per-tier rule moves into peerMaxConcurrency(), with a table test. The CRD field description (regenerated with make manifests, so the Helm chart copy matches) and the README say what an omitted maxConcurrency means. No CRD schema change.

With this, every peer of a chart-rendered route carries an explicit cap, and default_max only matters for ModelRoutes that leave the backend tier uncapped.

Testing

  • go test ./... passes.
  • New TestPeerMaxConcurrency covers these cases:
    • cart from backend, and cart static;
    • backend;
    • backend-svc derived, and following scaling (2 -> 4 backends);
    • an explicit backend-svc value winning;
    • no backend cap, and no backends.
  • TestReconcileGLM now expects the fallback peer as { "10.96.0.50", 8050, "backend-svc-0", 1, 200 }.
  • Control run on the old logic: both tests fail. TestReconcileGLM reports the missing ..., 1, 200 } line; the table test gets 0 instead of 200 / 400.

Rollout note

Upgrading changes the rendered conf of every route that has a backend-svc tier (the fifth column appears). That means one openresty reload per route, and no pod restarts.

The backend-svc tier is one Service VIP in front of the same backends as
the pod-IP tier, but it was rendered without a per-peer maximum unless the
ModelRoute set one, and no chart does. openresty then used its default_max
(20) for it. When every pod-IP peer is banned and traffic falls back to
this tier, the whole route was capped at 20 in-flight requests, while the
backends behind the VIP could take maxConcurrency x N.

When a backend-svc peer has no maxConcurrency, give it the backend tier's
maxConcurrency x number of backends, the way a cart peer with
maxConcurrencyFromBackend already gets it, so it follows scaling too. An
explicit maxConcurrency still wins. With no backend cap (or no backends)
the value stays 0 and openresty's default_max applies, as before.

The per-tier rule moves into peerMaxConcurrency, with a table test. The
reconcile test now expects the fallback peer at 100 x 2 = 200. The CRD
field description and README say what an omitted maxConcurrency means.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant