Repository navigation
fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest - #698
fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest#698deanq wants to merge 2 commits into
Conversation
celine-runpod
changed the title
celine-runpod
changed the title
|
bugbot run |
deanq
changed the title
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the changes aimed at reducing literal-secret exposure in deployment manifests.
No open issues 🎉
What changed in this PR
This PR reduces exposure of literal environment secrets in deployment manifests while preserving how workers receive their environment values.
Changes:
- Removes queue and API environment values from uploaded manifests, retaining reserved-name validation and task environment values needed by the backend.
- Strips all resource environment values from the packaged manifest without modifying the caller’s manifest.
- Warns about secret-looking literal environment values without logging their contents, with regression tests for filtering and warnings.
| File | Description |
|---|---|
tests/test_apps/test_deploy.py |
Adds coverage for environment filtering, task retention, reserved names, packaging, and secret warnings. |
runpod/apps/deploy.py |
Filters manifest environment values and warns about secret-looking literals. |
Review effort: Balanced
32527df to
30afb6d
Compare
| ) | ||
|
|
||
|
|
||
| def _without_env(manifest: Dict[str, Any]) -> Dict[str, Any]: |
There was a problem hiding this comment.
if you dont write env to queue and api manifests from their specs i dont think this function is needed anymore?
There was a problem hiding this comment.
It still earns its place, but only for tasks. Endpoints (queue/api) now go through to_manifest(include_env=False), so their env never enters the manifest at all. Tasks use include_env=True because the task env has to travel in the in-memory manifest passed to finalize_artifact_upload(...) (line 626), which is how the backend registers the task definitions (resourceSpec.tasks, validated just below).
_without_env is applied only to the copy baked into the artifact tarball (runpod_manifest.json, line 290), so that on-disk copy carries no task env. Drop it and task secrets land in plaintext inside the artifact, which is the exact leak this PR closes. Happy to add a short comment at the call site making that split explicit.
6cce11a to
b3f9d80
Compare
Deploy wrote every resource's env, literal values included, into the manifest it uploads and packs into the build artifact. - Queue and API env is checked against reserved names, then left out of the manifest. Endpoints get their env from the endpoint template. - Task env stays in the manifest sent to the backend, which launches deployed task pods from it. - The packaged runpod_manifest.json carries no env for any resource. - Warn when a secret-looking env name (*_TOKEN, *_KEY, *_SECRET, *_PASSWORD) has a literal value, pointing to runpod.Secret. SLS-994
Endpoint env was serialized into the manifest entry and then popped back out. Build the entry without it instead, via a new include_env flag on ResourceSpec.to_manifest, and validate the reserved keys from the spec. Behavior is unchanged: endpoints carry no env in the manifest, task env still goes to the backend.
b3f9d80 to
a410ada
Compare
What
rp flash deploywrote every resource'senvinto the build manifest, including literal values such asenv={"HF_TOKEN": os.environ["HF_TOKEN"]}. That manifest is sent to the backend when the build is finalized, and also packed into the artifact asrunpod_manifest.json. Neither copy needs the endpoint values.Changes:
envis still checked against reserved names, then left out of the manifest. Endpoints already get their env from the endpoint template, so workers see the same values.envstays in the manifest sent to the backend, because the backend launches deployed task pods from it.runpod_manifest.jsonhas noenvfor any resource. Nothing reads env from that file._TOKEN,_KEY,_SECRETor_PASSWORDhas a literal value, and points torp secret addplusrunpod.Secret(...). The warning names the variable, never the value.Why this shape
runpod.Secretvalues were already sent as references, so only literal values were affected. Literal values still have to reach the endpoint template or task definition, which is where they're used. The warning steers secret-looking values torunpod.Secret, the only way to keep a value out of those definitions.Test plan
tests/test_apps/test_deploy.py:runpod.Secretvalues and ordinary names don't warnmake quality-check(2783 passed)DEMO_TOKENandMODE. The CLI printed the warning, the worker returned both values, and the stored build manifest did not contain the literal value. A deploy of the same app from unfixed code stored the value in the manifest.