Skip to content

feat(modelsdev): add an offline switch for the models.dev catalog download - #4768

Draft
3431481073 wants to merge 2 commits into
docker:mainfrom
3431481073:feat/4714-offline-models-dev
Draft

3431481073 wants to merge 2 commits into
docker:mainfrom
3431481073:feat/4714-offline-models-dev

Conversation

@3431481073

Copy link
Copy Markdown

Summary

docker-agent run downloads the models.dev catalog on first use and caches it for 24 hours. Unlike telemetry, self-update, and tool auto-install, this outbound call had no opt-out, which is a problem for air-gapped installs, strict egress policies, and sandboxed runtimes that allocate a fresh HOME per session.

This adds two opt-out environment variables:

  • DOCKER_AGENT_MODELS_DEV=off (also false, 0, no)
  • DOCKER_AGENT_OFFLINE=1 (also true, yes, on)

When either is set, Store.getDatabase behaves as allowFetch=false: lookups resolve from the on-disk cache (including a stale one) and otherwise from the snapshot embedded at build time, with no request to models.dev. Default behavior is unchanged. A forced model-picker refresh is rejected in offline mode rather than silently going to the network.

Fixes #4714

Changes

  • Add WithOffline(bool) store option and read DOCKER_AGENT_MODELS_DEV / DOCKER_AGENT_OFFLINE in NewStore; an explicit option overrides the environment.
  • Force allowFetch=false in getDatabase when offline, reusing the existing cache-or-embedded fallback.
  • Guard Refresh against network access in offline mode.
  • Add tests for env parsing, cold-cache embedded fallback, stale-cache reuse, default online behavior, env wiring, option override, and offline refresh.
  • Document both variables in the runtime overrides table.

Testing

  • go test ./pkg/modelsdev/... -count=1 -v
  • go vet ./pkg/modelsdev/... ./pkg/config/... ./pkg/runtime/...
  • go test ./pkg/config/... ./pkg/runtime/... -count=1

task test and task lint were not run because task and golangci-lint are not installed in this environment; targeted tests and go vet pass.

Notes

modelsdev.Fetch remains an explicit live-fetch API (used by an examples test, not by docker-agent run) and is intentionally unaffected by the Store-level switch.

@3431481073
3431481073 requested a review from a team as a code owner October 29, 2026 18:12
@aheritier aheritier added area/config For configuration parsing, YAML, environment variables area/docs Documentation changes area/models LLM model integrations and model providers kind/feat PR adds a new feature (maps to feat:). Use on PRs only. status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key labels Oct 29, 2026
@aheritier

Copy link
Copy Markdown
Collaborator

👋 Some commits in this PR are not signed and verified by GitHub. Please sign your commits with a GPG or SSH key registered in your GitHub account, then force-push.

Commits that are not verified: 8269e6d

See GitHub\x27s guide on signing commits for setup instructions. I\x27ve added status/needs-signed-commits; it will be removed automatically once every commit in this PR carries a valid GitHub-verified signature.

@aheritier
aheritier marked this pull request as draft October 29, 2026 22:28
@3431481073
3431481073 force-pushed the feat/4714-offline-models-dev branch from 8269e6d to e22d269 Compare October 30, 2026 03:05
@3431481073
3431481073 marked this pull request as ready for review October 30, 2026 03:06
@3431481073

Copy link
Copy Markdown
Author

Signed the commit with an SSH signing key and folded in the lint fix (assert.Len). GitHub reports e22d269 as verified (reason: valid), and the PR is marked ready for review.

The new ci, docs, and codeql runs are action_required because this is a fork PR, so I cannot start them. Could you approve them when you get a chance? The status/needs-signed-commits label still shows on my side; if the signature check does not clear it after the workflow runs, could you remove it? Thanks.

@aheritier aheritier removed the status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key label Oct 30, 2026

@aheritier aheritier left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff and tests against #4714 and the background in #3165. See inline comment.

CI: verified green for e22d269. The previously missing ci / gate is now present and successful, with all seven dependencies successful; docs and CodeQL also passed. The Linux CI log confirms the modelsdev tests passed. Local tests were not run because this checkout is at a different commit and Go is unavailable.

Issue context: complete accessible descriptions and comments retrieved for #4714 and #3165, along with all PR comments, reviews, and threads. No Jira references found. The PR is not draft and has no merge conflicts; GitHub reports review required.

Comment thread pkg/modelsdev/store.go Outdated
fetch = fetchFromAPI
}

offline := offlineFromEnv()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Forward the opt-out variables into sandbox runs (confidence: high).

The failing case is DOCKER_AGENT_MODELS_DEV=off docker-agent run --sandbox agent.yaml with a normal built-in-provider config and a cold guest cache; DOCKER_AGENT_OFFLINE=1 has the same problem. This constructor reads the guest process environment, but cmd/root/sandbox.go:253-259 forwards only gathered credentials and the gateway setting. pkg/sandbox/sandbox.go:329-337 gathers model/evaluator/tool credentials, not either new variable. BuildExecCmd puts the host environment on the sandbox CLI process, but only its explicit -e arguments populate the guest environment. Consequently the guest constructs an online Store and can download the catalog despite the opt-out; allowSandboxHosts also still permits models.dev.

Forward both variables explicitly through the sandbox exec environment and add a regression test covering each switch across this boundary.

@aheritier
aheritier marked this pull request as draft October 31, 2026 18:16
Add DOCKER_AGENT_MODELS_DEV=off and DOCKER_AGENT_OFFLINE=1 to make
catalog lookups resolve from the on-disk cache or the embedded build-time
snapshot without contacting models.dev. The default behaviour is unchanged.

Refs docker#4714

Signed-off-by: xby <3431481073@qq.com>
The guest builds its own catalog Store from its process environment, and
only explicit -e values cross the sandbox boundary. Forward
DOCKER_AGENT_MODELS_DEV and DOCKER_AGENT_OFFLINE like the gateway, and stop
allowlisting models.dev in the sandbox proxy when the catalog is offline.

Adds regression tests for both switches at the sandbox boundary, an empty
guest cache falling back to the embedded snapshot, and the egress allowlist.

Refs docker#4714

Signed-off-by: xby <3431481073@qq.com>
@3431481073
3431481073 force-pushed the feat/4714-offline-models-dev branch from e22d269 to c942f23 Compare November 1, 2026 15:25
@3431481073
3431481073 marked this pull request as ready for review November 1, 2026 15:25
@3431481073

Copy link
Copy Markdown
Author

Addressed the sandbox forwarding in c942f23:

  • runInSandbox now forwards DOCKER_AGENT_MODELS_DEV and DOCKER_AGENT_OFFLINE as explicit -e values (and on the sandbox CLI environment). The store and the sandbox now share the same OfflineEnabled() parser.
  • Offline mode no longer adds models.dev to the sandbox proxy allowlist, and the printed allowance says so.
  • Tests cover each switch separately and together across the sandbox boundary, a cold guest cache falling back to the embedded snapshot, and the allowlist difference.
  • go test ./pkg/modelsdev/... ./cmd/root/ and go vet pass. Both commits are SSH-signed and DCO-signed; GitHub reports both as verified.

Could you take another look?

@3431481073

Copy link
Copy Markdown
Author

Both commits are SSH-signed and GitHub-verified. CI is action_required for the fork workflow; could you approve the runs when convenient?

@aheritier
aheritier self-requested a review November 4, 2026 12:09

@aheritier aheritier left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all seven changed files against #4714 and the background in #3165, including cache/embedded fallback, refresh rejection, environment parsing, and sandbox forwarding. See inline comment.

CI: verified green for c942f23, including gate, lint, Linux/Windows tests, docs, and CodeQL. The Linux test log confirms cmd/root and pkg/modelsdev passed. Local tests were not run because the checkout is at a different commit.

Issue context: complete accessible descriptions and comments retrieved for #4714 and #3165, along with all PR comments, reviews, and thread state. No Jira references found. The PR is not draft and has no merge conflicts; GitHub reports changes requested.

Prior threads: #4432766721 — partially addressed for local sandboxes; the same forwarding defect remains on the cloud path. The thread is unresolved and outdated, so the inline comment cites it at a current changed line.

Comment thread cmd/root/sandbox.go
// The guest builds its own catalog Store from its process environment, and
// only explicit -e values cross the sandbox boundary. Forward the offline
// switches explicitly for the same reason as the gateway.
offlineFlags, offlineVars := offlineSandboxEnv(os.Getenv)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Forward the offline switches through the cloud sandbox path too (confidence: high).

The local sandbox case from #4432766721 is fixed, but DOCKER_AGENT_MODELS_DEV=off docker-agent run --cloud --exec --model openai/gpt-5.6 default "hello" still loses the switch; DOCKER_AGENT_OFFLINE=1 has the same problem. runInSandbox returns into runInCloudSandbox at lines 122–123, before reaching this forwarding block. In cmd/root/sandbox_cloud.go:119–120, the cloud exec environment contains only SANDBOX_VM_ID and tracing values. pkg/sandbox/sandbox.go:263–289 forwards only the explicit exec environment flags into the guest, not the host CLI's entire environment. Consequently the guest's NewStore sees neither switch and, with a cold cache, attempts the catalog download instead of directly using the embedded snapshot. If cloud policy permits models.dev, the download succeeds despite the opt-out; otherwise it still makes the prohibited attempt.

Apply the forwarding helper to the cloud exec path as well, and add a regression test for each switch through that path. This would complete the earlier finding rather than leaving one supported sandbox mode outside the documented guarantee.

@aheritier
aheritier marked this pull request as draft November 4, 2026 20:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config For configuration parsing, YAML, environment variables area/docs Documentation changes area/models LLM model integrations and model providers kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A setting to turn off the models.dev catalog download (offline and strict-network use)

2 participants