Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions web/src/modules/docs/content/en-US/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,28 @@ Most fields can stay empty and take the site profile's and chart's defaults.

**Adaptive concurrency** groups the master switch, floor fraction, absolute floor, start point, idle hold, and TTFT decisions together; its internal knobs are available under **Advanced adaptive options**. Turning the switch off hides and clears the group's entered values; turning it back on does not restore them. **Default peer maximum** stays in **Capacity & failover** and still applies when adaptive concurrency is off; it also influences the static maximum used for adaptive bounds. Review the diff before confirming an upgrade.

## Video models

A variant whose engine is `video` (such as MiniMax H3, chart `minimax-h3`) uses the same form as an LLM: service ID, route, namespace, model path, GPU products, preset, tolerations, priority class, scheduler, inputs and the plan editor all work as above. The form adds the fields below and leaves out what only an LLM chart takes: replicas and scaler bounds (each pool has its own), route tuning, ServiceMonitor, image override and the compile cache.

| Field | Notes |
|---|---|
| Pools | FL2VA (text and first/last frame to video) and Ref2VA (reference media to video), each on or off with its own replicas, TP size and Ulysses degree. GPUs per pod = TP × Ulysses; the form shows the total |
| Turbo | Ref2VA only: the 8-step Turbo LoRA; off serves the base model |
| SLO | The form's SLO switch turns on a scaler and an SLO requirement for each pool. The chart creates each SLO with a minimum of 1 replica and **no maximum, so nothing is scaled yet** |
| Storage | The shared asset volume: its storage class must support ReadWriteMany. When the variant declares the class as an input, the site profile can answer it and the input field asks for it; otherwise empty uses the cluster default |
| Database URL | Optional; empty uses the chart's default. On an upgrade, empty keeps the release's own |

To scale a pool, open **Edit SLO** on the release once it is up. There is one section per pool:

- **Bounds**: the minimum (replicas or concurrency) and the maximum (replicas). An empty maximum is autoscaling off. A save only adds or changes values, so to turn a pool's scaling off again use **Reset**: it removes the maximum and sets the priority back to normal, and keeps the minimum and the queue settings. Set the priority again after a reset if the pool had it high.
- **Max queue depth**: queued and running tasks per replica before the pool gets another replica.
- **Priority**: high or normal.

A video release serves a job API rather than chat. Its access card shows how to submit a task (`POST /v2/video_generation`), poll it (`GET /v2/query/video_generation/<task_id>`) and download the result (`GET /v2/video_generation/<task_id>/content`, no key needed). Its **Check** runs a readiness probe of the route's `/readyz`, which answers only when the router, a server and the database behind it are all up. It is not offered in the Playground.

Download links point back at H3. When the site profile names a `route.gateway`, the gateway URL for the release's route is filled in for the chart automatically. An explicit `server.config.publicBaseUrl` in the Override YAML still wins.

## Generation recovery

Enable **Routing → Generation recovery** in the deploy or upgrade form. Enter the **Actual model** whose weights are deployed, such as `kimi-k3` or `glm-5.3`. This selects the continuation strategy; it is not the client-facing served name.
Expand Down
22 changes: 22 additions & 0 deletions web/src/modules/docs/content/zh-CN/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,28 @@

**自适应并发** 的总开关、下限比例、绝对下限、起始并发、空闲保持和 TTFT 判断位于同一区域,内部参数可从 **高级自适应参数** 添加。关闭后隐藏本组输入并清空已填写参数,再开启时不会恢复这些旧值。**默认节点并发上限** 留在 **容量与故障切换** 中,关闭自适应并发时也有效;它也影响自适应上下限使用的静态最大值。升级时仍需先检查差异再确认。

## 视频模型

引擎为 `video` 的变体(如 MiniMax H3,chart `minimax-h3`)使用和 LLM 相同的部署表单:服务 ID、路由、命名空间、模型路径、GPU 型号、预设、容忍、优先级类、调度器、输入项和计划编辑器都和上文一样。表单另外增加下列字段,并去掉只有 LLM chart 才有的设置:副本数与扩缩上下限(每个池单独设置)、路由调优、ServiceMonitor、镜像覆盖和编译缓存。

| 字段 | 说明 |
|---|---|
| 池 | FL2VA(文生视频、首尾帧生视频)与 Ref2VA(参考素材生视频)各自开关,分别设置副本数、TP 并行度和 Ulysses 并行度。每个 Pod 的 GPU 数 = TP × Ulysses,表单会显示总数 |
| Turbo | 仅 Ref2VA:8 步 Turbo LoRA,关闭则使用基座模型 |
| SLO | 表单的 SLO 开关为每个池启用扩缩器和 SLO 要求。chart 为每个池创建一个 SLO,最小副本数为 1、**没有最大副本数,因此此时还不会扩缩** |
| 存储 | 共享素材盘:存储类必须支持 ReadWriteMany。变体把存储类声明为输入项时,站点 profile 可以给出答案,由输入项字段填写;否则留空使用集群默认 |
| 数据库 URL | 可选,留空使用 chart 默认值;升级时留空保留 release 原有的值 |

要让某个池自动扩缩,在 release 就绪后打开 **编辑 SLO**,每个池一节:

- **副本数上下限**:最小值(副本数或并发数)与最大值(副本数)。最大值留空即不自动扩缩。保存只会新增或修改字段,要重新关闭某个池的扩缩,请用 **重置**:它会去掉最大副本数并把优先级恢复为普通,最小值和队列设置保留。该池原来是高优先级的话,重置后需要重新设置。
- **最大队列深度**:每个副本排队和运行中的任务数超过该值时扩容。
- **优先级**:高或普通。

视频 release 提供的是任务接口而非对话接口:访问卡片给出提交任务(`POST /v2/video_generation`)、查询状态(`GET /v2/query/video_generation/<task_id>`)和下载结果(`GET /v2/video_generation/<task_id>/content`,无需 key)的示例。**检查** 会探测路由的 `/readyz`,只有路由器、server 和后面的数据库都就绪时才会通过。它不会出现在 Playground 中。

下载链接指回 H3 本身。站点配置设置了 `route.gateway` 时,release 路由对应的网关地址会自动传给 chart;在覆盖 YAML 中显式设置的 `server.config.publicBaseUrl` 仍然优先。

## 生成中断恢复

在部署或升级表单的 **路由 → 生成中断恢复** 中开启,填写权重对应的 **实际模型**,例如 `kimi-k3` 或 `glm-5.3`。它用于选择续写策略,不是客户端调用时使用的服务模型名。
Expand Down
9 changes: 6 additions & 3 deletions web/src/modules/inferences/InferenceDetail.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { useReleaseObjects } from "@swiss/components/ReleaseObjects";
import { ReleaseObjects } from "@swiss/components/ReleaseObjects";
import { Endpoint } from "@swiss/components/Endpoint";
import { Provenance } from "@swiss/components/Provenance";
import { isVideo, jobSLOs } from "@swiss/lib/video";
import { useT } from "@/modules/inferences/i18n";
import { modelLine, parseTab, releaseState, sloBlocked, visibleTabs, type Tab } from "@/modules/inferences/lib";
import { statusOf } from "@/modules/inferences/components/StatusDot";
Expand Down Expand Up @@ -112,7 +113,9 @@ export function InferenceDetail() {
const source = plan.data?.source;
const model = modelLine(source ?? {}) || s.model;
const pods = s.pods.map((p) => p.name);
const blocked = sloBlocked(s, plan.data);
// A video release has one SLO per pool; the sheet edits each.
const video = isVideo(plan.data?.engine);
const blocked = sloBlocked(s, plan.data, video && objects.data ? jobSLOs(objects.data.objects).length > 0 : undefined);
const sloOpen = !removing && (editingSLO || (sloSetup && blocked !== "off" && s.planStatus?.phase !== "failed"));
const closeSLO = () => {
setEditingSLO(false);
Expand Down Expand Up @@ -176,7 +179,7 @@ export function InferenceDetail() {
<Events namespace={namespace} release={release} pods={pods} />
</TabsContent>
<TabsContent value="resources">
<ReleaseObjects namespace={namespace} release={release} />
<ReleaseObjects namespace={namespace} release={release} video={video} />
</TabsContent>
<TabsContent value="check">
<Endpoint namespace={namespace} release={release} status={s} access={false} />
Expand All @@ -196,7 +199,7 @@ export function InferenceDetail() {
)}
</Tabs>

<SLOSheet target={sloOpen ? { namespace, release } : null} canEdit={!readOnly && !removing && !external} setup={sloSetup} waiting={blocked === "pending"} onClose={closeSLO} />
<SLOSheet target={sloOpen ? { namespace, release } : null} video={video} canEdit={!readOnly && !removing && !external} setup={sloSetup} waiting={blocked === "pending"} onClose={closeSLO} />
<DeploySheet target={upgrading ? { kind: "upgrade", namespace, release } : null} onClose={() => setUpgrading(false)} />
<UninstallDialog
target={uninstalling ? { namespace, release } : null}
Expand Down
95 changes: 58 additions & 37 deletions web/src/modules/inferences/components/DeployForm.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import { FALLBACK_ROUTE_OPTIONS, routeDirectives, runtimeValue, type RouteDirect
import { useT } from "@/modules/inferences/i18n";
import { isAdaptiveOption, toggleAdaptive } from "../adaptive-concurrency";
import { OverrideYaml, type HelmValuesSource } from "./OverrideYaml";
import { VideoPools, VideoStorage, declaresStorageClass } from "./VideoFields";
import { NAMED_VALUES, extraValue, fracOutOfRange, pickTaints, pickedTaints, serviceIdError, type ClusterTaint, type Section } from "@/modules/inferences/deploy-lib";

const EFFECTS = ["", "NoSchedule", "PreferNoSchedule", "NoExecute"];
Expand Down Expand Up @@ -99,11 +100,14 @@ export interface DeployFormProps {
presets?: Record<string, Preset>;
// The taints the cluster's nodes carry, to pick tolerations from.
taints?: ClusterTaint[];
// A video variant: the form adds the video chart's own fields (form.video)
// and leaves out what only an LLM chart takes -- route tuning, the compile
// cache and the engine image.
video?: boolean;
}

export function DeployForm({ form, onChange, upgrade, disabled, submitted, target, cluster, defaults, gpuOptions, gpuListed, gpuResources, capacity, expanded, onExpand, idPrefix, routeOptions, catalogRoute, helmValues, onHelmValuesOpenChange, slots, slotsError, resources, pd, nodes, gpuSpecs, gpuVendor, recovery, inputs, droppedInputs, remember, onRemember, presets, taints }: DeployFormProps) {
export function DeployForm({ form, onChange, upgrade, disabled, submitted, target, cluster, defaults, gpuOptions, gpuListed, gpuResources, capacity, expanded, onExpand, idPrefix, routeOptions, catalogRoute, helmValues, onHelmValuesOpenChange, slots, slotsError, resources, pd, nodes, gpuSpecs, gpuVendor, recovery, inputs, droppedInputs, remember, onRemember, presets, taints, video }: DeployFormProps) {
const t = useT();
const checked = useDebounced(form.inputs, CHECK_MS);
const on = effective(form);
const f = (k: string) => t(`deploy.fields.${k}`);
const forced = unlistedProducts(gpuListed, form.gpuProducts);
Expand Down Expand Up @@ -167,15 +171,17 @@ export function DeployForm({ form, onChange, upgrade, disabled, submitted, targe
</FloatingField>
)}
</div>
<FloatingField layout="inline" label={f("slo")} hint={f("sloHint")}>
<FloatingField layout="inline" label={f("slo")} hint={video ? t("deploy.video.autoscaleHint") : f("sloHint")}>
{sw("slo", f("slo"))}
</FloatingField>
{video && <VideoStorage value={form.video} onChange={(v) => onChange({ video: v })} upgrade={upgrade} classInput={declaresStorageClass(inputs)} />}
</>,
)}

{section(
"resources",
<>
{video && <VideoPools value={form.video} onChange={(v) => onChange({ video: v })} submitted={submitted} />}
{gpuResources && <GpuResource resources={gpuResources} value={form.gpuResource} onChange={(gpuResource) => onChange({ gpuResource })} />}
{(!pd?.enabled || pd.colocate || form.gpuProducts.length > 0) && (
<FloatingField label={f("gpuProducts")} hint={f("gpuProductsHint")} notice={forced.length > 0 ? t("deploy.fields.gpuForced", { products: forced.join(", ") }) : undefined}>
Expand Down Expand Up @@ -238,39 +244,14 @@ export function DeployForm({ form, onChange, upgrade, disabled, submitted, targe
</div>
</div>
)}
{inputs?.map((row) => {
const { input, answer, choices, site } = row;
const value = form.inputs[input.key] ?? "";
const error = inputError({ input, answer }, checked[input.key]);
const set = (v: string) => onChange({ inputs: { ...form.inputs, [input.key]: v } });
return (
<div key={input.key} className="flex flex-col gap-1.5">
<FloatingField
value={shown(value, answer)}
label={input.title}
required
hint={input.description ? `${input.description} (${input.key})` : input.key}
aside={t("deploy.fields.inputExample", { example: String(input.example) })}
error={error && (error === "required" ? t("deploy.fields.inputRequired") : t(`deploy.fields.inputProblem.${input.type ?? "string"}`, { pattern: input.pattern ?? "" }))}
>
<Suggest value={value} onChange={set} suggestion={answer} placeholder={answer} choices={choices?.map((c) => ({ value: c.answer, label: f(`inputFrom.${c.from}`) }))} />
</FloatingField>
{remember && onRemember && rememberable(row, checked[input.key]) && (
<label className="flex flex-wrap items-center gap-1.5 px-1 text-xs text-muted-foreground">
<Checkbox checked={!!remember[input.key]} onCheckedChange={(on) => onRemember(input.key, !!on)} />
{f("inputRemember")}
<FieldHint>{f("inputRememberHint")}</FieldHint>
{site && <span className="break-all">· {t("deploy.fields.inputReplaces", { answer: site.written })}</span>}
</label>
)}
</div>
);
})}
<ImageRows form={form} onChange={onChange} image={defaults.image} mirror={defaults.imageMirror} slots={slots} error={slotsError} />
{inputs && <InputFields inputs={inputs} values={form.inputs} onChange={(v) => onChange({ inputs: v })} remember={remember} onRemember={onRemember} />}
{/* A video chart has no engine `image` path: before a dry run lists its
slots there is no row to guess, so it shows only the chart's own. */}
<ImageRows form={form} onChange={onChange} image={video ? undefined : defaults.image} mirror={defaults.imageMirror} slots={slots} error={slotsError} />
</>,
)}

{section(
{!video && section(
"routing",
<>
{recovery}
Expand Down Expand Up @@ -319,7 +300,7 @@ export function DeployForm({ form, onChange, upgrade, disabled, submitted, targe
{section(
"advanced",
<>
{cache && (
{cache && !video && (
<FloatingField
layout="inline"
label={f("cache")}
Expand Down Expand Up @@ -681,6 +662,46 @@ function ImageRows({
);
}

// The variant's catalog inputs, one field each. Shared with the video form.
export function InputFields({ inputs, values, onChange, remember, onRemember }: { inputs: InputRow[]; values: Record<string, string>; onChange: (v: Record<string, string>) => void; remember?: Record<string, boolean>; onRemember?: (key: string, on: boolean) => void }) {
const t = useT();
const f = (k: string) => t(`deploy.fields.${k}`);
const checked = useDebounced(values, CHECK_MS);
const shown = (value: string, fallback?: string) => value || fallback || "";
return (
<>
{inputs.map((row) => {
const { input, answer, choices, site } = row;
const value = values[input.key] ?? "";
const error = inputError({ input, answer }, checked[input.key]);
const set = (v: string) => onChange({ ...values, [input.key]: v });
return (
<div key={input.key} className="flex flex-col gap-1.5">
<FloatingField
value={shown(value, answer)}
label={input.title}
required
hint={input.description ? `${input.description} (${input.key})` : input.key}
aside={t("deploy.fields.inputExample", { example: String(input.example) })}
error={error && (error === "required" ? t("deploy.fields.inputRequired") : t(`deploy.fields.inputProblem.${input.type ?? "string"}`, { pattern: input.pattern ?? "" }))}
>
<Suggest value={value} onChange={set} suggestion={answer} placeholder={answer} choices={choices?.map((c) => ({ value: c.answer, label: f(`inputFrom.${c.from}`) }))} />
</FloatingField>
{remember && onRemember && rememberable(row, checked[input.key]) && (
<label className="flex flex-wrap items-center gap-1.5 px-1 text-xs text-muted-foreground">
<Checkbox checked={!!remember[input.key]} onCheckedChange={(on) => onRemember(input.key, !!on)} />
{f("inputRemember")}
<FieldHint>{f("inputRememberHint")}</FieldHint>
{site && <span className="break-all">· {t("deploy.fields.inputReplaces", { answer: site.written })}</span>}
</label>
)}
</div>
);
})}
</>
);
}

function CacheFields({ form, onChange, defaults }: { form: Form; onChange: (patch: Partial<Form>) => void; defaults: CacheDefaults }) {
const t = useT();
const f = (k: string) => t(`deploy.fields.${k}`);
Expand Down Expand Up @@ -776,7 +797,7 @@ function GpuResource({ resources, value, onChange }: { resources: string[]; valu
);
}

function Tolerations({ rows, taints, onChange }: { rows: Toleration[]; taints: ClusterTaint[]; onChange: (v: Toleration[]) => void }) {
export function Tolerations({ rows, taints, onChange }: { rows: Toleration[]; taints: ClusterTaint[]; onChange: (v: Toleration[]) => void }) {
const t = useT();
const f = (k: string) => t(`deploy.fields.${k}`);
const patch = (i: number, p: Partial<Toleration>) => onChange(rows.map((r, j) => (j === i ? { ...r, ...p } : r)));
Expand Down Expand Up @@ -808,7 +829,7 @@ function Tolerations({ rows, taints, onChange }: { rows: Toleration[]; taints: C

const NO_PRESET = "__none";

function PresetField({ presets, value, onChange }: { presets: Record<string, Preset>; value: string; onChange: (v: string) => void }) {
export function PresetField({ presets, value, onChange }: { presets: Record<string, Preset>; value: string; onChange: (v: string) => void }) {
const t = useT();
const f = (k: string) => t(`deploy.fields.${k}`);
const picked = presets[value];
Expand Down Expand Up @@ -959,7 +980,7 @@ export function RouteExtras({ form, onChange, options, showRuntime = false, mode

// Tab fills an empty field with the default it shows, as swiss's own form does.
// choices list under the field when the empty field is clicked; typing closes the list.
function Suggest({ value, onChange, suggestion, bare, placeholder, className, choices, ...props }: Omit<React.ComponentProps<typeof Input>, "value" | "onChange"> & { value: string; onChange: (v: string) => void; suggestion?: string; bare?: boolean; choices?: { value: string; label: string }[] }) {
export function Suggest({ value, onChange, suggestion, bare, placeholder, className, choices, ...props }: Omit<React.ComponentProps<typeof Input>, "value" | "onChange"> & { value: string; onChange: (v: string) => void; suggestion?: string; bare?: boolean; choices?: { value: string; label: string }[] }) {
const [focused, setFocused] = useState(false);
const [listed, setListed] = useState(false);
const offer = !value && !props.disabled ? suggestion : undefined;
Expand Down
Loading
Loading