Repository navigation
Derive the backend-svc peer cap from the backend tier - #10
Open
zhanghaohit wants to merge 1 commit into
Open
zhanghaohit wants to merge 1 commit into
zhanghaohit wants to merge 1 commit into
Conversation
The backend-svc tier is one Service VIP in front of the same backends as the pod-IP tier, but it was rendered without a per-peer maximum unless the ModelRoute set one, and no chart does. openresty then used its default_max (20) for it. When every pod-IP peer is banned and traffic falls back to this tier, the whole route was capped at 20 in-flight requests, while the backends behind the VIP could take maxConcurrency x N. When a backend-svc peer has no maxConcurrency, give it the backend tier's maxConcurrency x number of backends, the way a cart peer with maxConcurrencyFromBackend already gets it, so it follows scaling too. An explicit maxConcurrency still wins. With no backend cap (or no backends) the value stays 0 and openresty's default_max applies, as before. The per-tier rule moves into peerMaxConcurrency, with a table test. The reconcile test now expects the fallback peer at 100 x 2 = 200. The CRD field description and README say what an omitted maxConcurrency means.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
backend-svctier is a single Service VIP in front of the same backends as the pod-IPbackendtier. It was rendered without a per-peer maximum unless the ModelRoute setmaxConcurrencyon it, and the sglang / vllm charts don't. openresty then used itsdefault_max(20) for that peer. A rendered route looked like this:Normally that tier carries nothing. But when every cart and pod-IP peer is banned and traffic falls back to the VIP, the whole route is capped at 20 in-flight requests, while the backends behind it can take
maxConcurrency x N.Change
backend-svcpeer without its ownmaxConcurrencygets the backend tier'smaxConcurrencyx number of backends, the same rule acartpeer withmaxConcurrencyFromBackendalready follows. It tracks scaling too.maxConcurrencyonbackend-svcis used as-is.default_maxapplies, as before.The per-tier rule moves into
peerMaxConcurrency(), with a table test. The CRD field description (regenerated withmake manifests, so the Helm chart copy matches) and the README say what an omittedmaxConcurrencymeans. No CRD schema change.With this, every peer of a chart-rendered route carries an explicit cap, and
default_maxonly matters for ModelRoutes that leave the backend tier uncapped.Testing
go test ./...passes.TestPeerMaxConcurrencycovers these cases:TestReconcileGLMnow expects the fallback peer as{ "10.96.0.50", 8050, "backend-svc-0", 1, 200 }.TestReconcileGLMreports the missing..., 1, 200 }line; the table test gets0instead of200/400.Rollout note
Upgrading changes the rendered conf of every route that has a
backend-svctier (the fifth column appears). That means one openresty reload per route, and no pod restarts.