You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support bring-your-own LLM (hosted OpenAI-compatible / Azure OpenAI) for language model integration #2034
Dev Proxy's language model integration is currently designed for local SLMs (default: Ollama llama3.2 at localhost:11434). As inference prices drop, using hosted LLMs is increasingly viable and yields better results (e.g. operation IDs/descriptions in OpenApiSpecGeneratorPlugin/TypeSpecGeneratorPlugin, mock responses in OpenAIMockResponsePlugin).
Current limitations
OpenAILanguageModelClient sends no Authorization/api-key header, so hosted endpoints (OpenAI, Azure OpenAI, GitHub Models, OpenRouter, Foundry) can't be used.
Only Ollama and OpenAI clients; no Azure OpenAI support (api-key header, api-version, deployment-based URLs).
IsEnabledCoreAsync sends a real "Are you there?" completion on startup, which costs money on paid endpoints.
languageModel schema describes the feature as "local language model".
Proposed changes
Add apiKey to languageModel config, supporting @env:VAR_NAME so secrets stay out of config files; send as Bearer token (OpenAI-compatible) or api-key (Azure).
Add AzureOpenAI client (or evaluate moving to Microsoft.Extensions.AIIChatClient).
Replace the startup availability probe with a non-billable check (e.g. GET /models) where possible.
Update rc.schema.json (current version folder) and docs.
Never log API keys.
Follow-ups (out of scope here): per-plugin model overrides, redaction/opt-in for sending captured traffic to cloud models.
Activity