Repository navigation
feat(web): deploy video models with the LLM form plus their own fields - #129
Draft
zhanghaohit wants to merge 15 commits into
Draft
zhanghaohit wants to merge 15 commits into
zhanghaohit wants to merge 15 commits into
Conversation
A catalog variant with engine "video" (first one: MiniMax H3 on the minimax-h3 chart) now opens a video deploy form instead of the LLM one, in both the console's deploy sheet and swiss's standalone pages. The LLM form and its requests are unchanged. - The video form writes the chart's pool, Turbo, storage, database and autoscaling values. Every key it writes lives in swiss/lib/videoChart.ts. - Upgrades of a video release use the same form, seeded from its plan. The database URL is never read back into the form; empty keeps it. - Video releases show the job API (submit, poll, download) instead of the chat example and checks, have no SLO action or panel, and are left out of the Playground.
Per-pool autoscaling goes to the pool's JobSLORequirement
(pools.<p>.sloRequirement.{minReplicas,maxReplicas,queue.maxDepth}), the
LLMScaler bounds keep the chart's, and a custom route is written as
modelRoute.nginx.route, which the chart and swissd both read before
serviceId. An empty route field now writes nothing. The deploy docs say how
server.config.publicBaseUrl takes __ROUTE__ for a site-wide setting.
Autoscaling follows the PD pattern. The deploy form has one Autoscale / SLO switch (scaler.enabled + sloRequirement.enabled) and no per-pool fields. The chart gives each pool a JobSLORequirement with no maximum, which is not scaled until the SLO card sets one. - The SLO card for a video release has one section per pool: min and max (empty max = autoscaling off), queue max depth, priority, and the queue query read-only. Each pool's service id comes from the release's JobSLORequirement objects and is sent to swissd's /slo as ?serviceId=. - The release page's Check is a readiness probe of the route's /readyz again, next to the job API examples. - publicBaseUrl guidance is gone from the form: swissd derives it from the site's route.gateway. The docs say so.
…on upgrade - pools.<p>.sglang.gpuCount = tp × ulysses for every enabled pool. swissd sends model.gpus as the variant's requires.gpus, and the chart refuses a render where the pod's GPU count is not tp × ulysses. Without this, changing the parallelism in the form broke the deploy. - An upgrade lays the form over the release's form layer, so a field the form leaves out keeps its old value. Emptying the route, storage class or size, priority class, scheduler or GPU products now sends "" (or []) when the release had set it. database.url is not cleared: empty there means keep.
An existing PVC's storage class cannot change and its size cannot shrink, and the chart renders an empty size as an invalid PVC. So an empty storage class or size on upgrade now keeps the release's value, like database.url, instead of sending a clear. The upgrade form locks the class and says the size can only grow. Route, GPU products, priority class and scheduler still clear when emptied.
A video variant used a form of its own, which repeated the LLM form's rows and drifted from them (no model path, scheduling written under other keys). It now shares the LLM form and its plan request: the form adds the video chart's pools, Turbo, shared volume and database, and leaves out what only an LLM chart takes (replicas and scaler bounds, route tuning, ServiceMonitor, the engine image, the compile cache). The SLO switch turns on the pools' scalers and JobSLOs. Scheduling goes to the top-level keys an LLM deploy writes, which the chart applies to the GPU pods; tolerations stay under sglang, the only place the chart reads them. An upgrade clears the sglang.* scheduling keys earlier forms wrote, and the storage class is asked by the variant's input when it declares one.
Edit SLO was greyed out for a video release whose form layer never named the SLO switch, though the chart's default had created its JobSLOs: for a video release the button now follows the JobSLOs that exist. The scaler step showed a pool with no maximum as failing, since the decision server publishes nothing for an unscaled pool and its LLMScaler reports that as unavailable: the step now reads only the pools that are scaled, and is off when none is.
…et keeps The SLO write answers with the fields it took, so the card lost the chart's queue query until the next read: it reads the whole SLO back now. Reset goes to the CRD defaults, which is normal priority rather than the chart's high; the button and the docs say so.
… zero in a video pool The deploy sheet rebuilt its whole form whenever the variant's values changed, so picking another version of an LLM variant threw away what was typed. The video fields are now seeded into the current form instead, once per video variant's values, and taken back off (GPU tolerations included) on leaving a video variant, on both deploy pages. An enabled pool with 0 replicas, TP or Ulysses is refused before compose.
…d clear an emptied route An upgrade read each switch from the release's form layer alone, so a release whose route, scaler, SLO or ServiceMonitor came from the site or the defaults opened with that switch off, and an upgrade made unchanged turned it off. The form layer still decides when it names a switch; otherwise the last layer that sets it does. An emptied custom route is sent as an empty route, so the overlay drops the old one and the route falls back to the service ID. A video release's shared volume size reads back empty when its form layer has none, which keeps the volume rather than writing the default over a larger one.
…O from its pools The console detail page already left unscaled pools out of the scaler step and offered Edit SLO from the live JobSLOs; the standalone swiss page did not. Both now share the selection of the scaled pools' scalers.
The PromQL the decision server reads a pool's backlog with is the chart's, and the card is for the people who run the service: it shows the bounds, the queue depth and the priority, and no longer the query.
The storage size and database fields read "kept", which did not say what was kept. An upgrade now reads the release's shared volume size and class from every plan layer and shows them: "currently 500Gi; empty keeps it", and the locked class shows the class itself. The database field says it keeps the current setting, and the deploy hint says an empty URL gives the service a database of its own.
Conflicts were the video form meeting main's image slots, kyaml input keys, per-role resources and cluster taints. Resolved to main's behaviour, with the video path kept: - Images: the video form shows the Images section too, from the chart's own slots only; the LLM engine's `image` fallback row is not offered for it. - Upgrade seeding goes through main's formFromPlan(plan, gpuSpecs, variant); the video branch reads back inputs, images and form affinity the same way. - The release's form layer is no longer laid under a new request (main's "the new layer is the form alone"); the video test now asserts that. - Inputs stay in the shared InputFields component, with main's note for an answer recorded on the release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
What this PR does and why
Catalog variants can now carry
engine: video(first user: MiniMax H3, chartminimax-h3; swissd accepts it in a parallel change). A video variant deploys with the same form as an LLM, plus the video chart's own fields (pools, Turbo, shared volume, database). The form leaves out what only an LLM chart takes: replicas and scaler bounds, route tuning, ServiceMonitor, the engine image, the compile cache.inferences)DeployFormwithvideo: addsVideoStorage(Basic) andVideoPools(Resources), no Routing sectionDeployForm, unchangedDeploySettingswithvideo: adds theVideoSettingscards, no Routing options cardDeploySettings, unchangedForm.video+planRequest(..., { video: true })Form+planRequest, unchangedVideoSLOCard: one section per poolSLOCard, unchanged/readyz)The type comes from
variant.engineon deploy andplan.engineon upgrade and detail pages. List rows useDeployment.engine(modelsphere/swiss#59), falling back to the chart name for an older swissd: swissd only allows LLM engines on thesglang/vllmcharts.Autoscaling follows the PD pattern. The form's existing SLO switch turns on the pools' scalers and SLOs. The chart then gives each pool a JobSLORequirement with its default minimum and queue depth and no maximum, so nothing is scaled. Scaling is set after the deploy in the SLO card.
Chart values the deploy form writes
Matched against modelsphere/helm-charts#105. Every key is in
web/src/modules/swiss/lib/videoChart.ts.pools.<p>.enabled(fl2va,ref2va)pools.<p>.replicaCountscaler.enabled)pools.<p>.sglang.args.tpSize/.ulyssesDegreepools.<p>.sglang.gpuCountmodel.gpusand refuses a render where they differ)pools.ref2va.turbo.enabledscaler.enabled+sloRequirement.enabledsharedStorage.storageClasssharedStorage.sizedatabase.urlmodelRoute.enabledroute.enabledmodelRoute.nginx.routesglang.tolerationsnvidia.com/gputolerationpriorityClassName/schedulerName/affinityUpgrade. An upgrade lays the form over the release's form layer, so a key the form leaves out keeps its old value, as for an LLM. A switch the form layer does not name (a release deployed through the API, or by an older form) is seeded from the last plan layer that sets it rather than a hard-coded default, for LLM and video alike, and an emptied custom route is sent as
""so it falls back to the service ID. Earlier preview builds wrotesglang.priorityClassName/.schedulerName/.gpuProduct; the form reads them back and an upgrade clears them, so the top-level keys apply.database.url,sharedStorage.storageClassandsharedStorage.sizeare never cleared: empty there means keep. An existing PVC's class cannot change and its size cannot shrink, and an empty size renders an invalid PVC. On upgrade the form locks the storage class and says the size can only grow.Not written: any
pools.<p>.sloRequirement.*orpools.<p>.scaler.*(the chart's defaults apply), andserver.config.publicBaseUrl(swissd derivesmodelRoute.urlfrom the site'sroute.gateway, and the chart uses it as the default).swissd API used
GET /api/releases/{ns}/{release}/objectsJobSLORequirementobjects: one per pool,spec.serviceId,minimumDeployment,maximumDeployment,queueGET/PUT/DELETE /api/releases/{ns}/{release}/slo?serviceId=<id>highPriority,minimumDeployment,maximumDeploymentonly when set,queue.maxDepth), reset one pool's JobSLO. WithoutserviceId, LLM calls are unchangedPOST /api/releases/{ns}/{release}/probe<url>/readyzfor a video releaseGET /api/deploymentsengineon each row (swiss#59)Shared pieces were exported rather than copied:
Row,Suggest,PathInput,Tolerations,Productsand an extractedInputFieldsfrom swissDeploySettings;RowandSelectfromSLOCard;Suggest,Tolerations,PresetFieldand an extractedInputFieldsfromDeployForm. The LLM forms and SLO card render as before.Which issue(s) this PR fixes
Refs modelsphere/helm-charts#105, modelsphere/swiss#59.
How it was tested
swiss/routes/Deploy.test.tsxandinferences/components/VideoFields.test.tsx: a video variant shows the LLM form's shared rows plus the video fields and none of the LLM-only ones; the storage class field gives way to a declared input; the LLM form is unchanged.swiss/lib/video.test.ts:planRequestwithvideowrites the shared keys and the video keys and nothing LLM-only;gpuCount= tp × ulysses; empty keeps the DB URL and volume; the legacysglang.*scheduling keys are cleared;formFromPlanround-trips a video release (and reads the legacy keys); GPU totals; list-row detection.swiss/components/VideoSLOCard.test.tsxcovers JobSLO discovery from objects,?serviceId=paths, edit bodies (no max when empty), validation, and a render per pool without TTFT/OTPS.swiss/components/VideoEndpoint.test.tsxcovers the job API examples and the readiness check vs the LLM checks.playground/targets.test.tscovers that video releases are left out.swiss/components/VideoSLOCard.test.tsxandinferences/lib.test.tsalso cover Edit SLO following a video release's live JobSLOs, and the scaler step reading only the pools with a maximum (an unscaled pool's "no decision" is not a failure).npm run typecheck,npm test(651 tests;shell/preferences.test.tsfails to load on this Node with or without this change),npm run build,npm run build:swiss./readyz, a T2V and a Ref2VA task each completed and downloaded through the derived public URL, and per-pool SLO edits, saves and resets from the console reached the JobSLOs and the decision server.Checks
npm run typecheck && npm test && npm run buildinweb/docs/content/{en-US,zh-CN}/deploy.md, "Video models")Notes for reviewers
database.urllands in the plan's form layer, which is stored in a ConfigMap beside the release. Keeping it in a Secret would need a swissd change.Release note