Creating and Managing Model Proxies
Model Proxy provides a unified access layer for multiple Large Language Model (LLM) providers. Model Proxies are deployed to Omni Gateway to enable governance, intelligent routing, and cost management for AI applications.
Model Proxy is supported on Managed Omni Gateway and Self-Managed Omni Gateway running in Connected Mode.
By creating a proxy, the user defines a singular LLM service that can receive requests for multiple LLM providers. This simplifies the developer experience. You can seamlessly add new models to the service without changing the endpoint.
Supported Endpoint Formats
Model Proxy supports these endpoint formats that are selected when creating a Model Proxy and can’t be changed later:
-
OpenAI: The OpenAI API format works with all supported LLM providers and supports multi-routing and fallback mechanisms.
-
Gemini: The native Gemini API format for direct Google Gemini model requests. Doesn’t support multi-routing or fallback mechanisms.
-
Anthropic: The native Anthropic API format for direct Anthropic Claude model requests. Doesn’t support multi-routing or fallback mechanisms.
Depending on configuration, the proxy then sends the request to the model defined by the user or dynamically sends the request to the provider that best matches the request:
-
Model-Based Routing: Static routing. The user specifies what model the Model Proxy should send the request to.
-
Semantic Routing: Dynamic routing. The Model Proxy chooses which model to send the request to based on the request content.
Supported LLM Providers
Model Proxy supports these LLM Providers and API endpoints:
| LLM Provider | Model | /chat/completions |
/responses |
|---|---|---|---|
OpenAI and Azure OpenAI |
gpt-5.2 |
Yes |
Yes |
gpt-5.2-pro |
Yes |
Yes |
|
gpt-5-mini |
Yes |
Yes |
|
gpt-5.2-codex |
Yes |
Yes |
|
gpt-5-nano |
Yes |
Yes |
|
gpt-5 |
Yes |
Yes |
|
gpt-4.1 |
Yes |
Yes |
|
gpt-4o-mini |
Yes |
Yes |
|
Gemini |
gemini-3-flash-preview |
Yes |
Yes |
gemini-2.5-flash |
Yes |
Yes |
|
gemini-2.5-flash-preview-09-2025 |
Yes |
Yes |
|
gemini-2.5-flash-lite |
Yes |
Yes |
|
Anthropic and Bedrock Anthropic |
Claude Sonnet 4.6 |
Yes |
Yes |
Claude Opus 4.6 |
Yes |
Yes |
|
Claude Opus 4.5 |
Yes |
Yes |
|
Claude Haiku 4.5 |
Yes |
Yes |
|
Claude Sonnet 4.5 |
Yes |
Yes |
|
Claude Opus 4 |
Yes |
Yes |
|
Claude Sonnet 4 |
Yes |
Yes |
|
Claude Sonnet 3.7 |
Yes |
Yes |
|
Claude Sonnet 3.5 |
Yes |
Yes |
|
Claude Haiku 3.5 |
Yes |
Yes |
|
NVIDIA Nemotron |
Nemotron 3 Nano 30B A3B |
Yes |
Yes |
Nemotron 3 Super 120B A12B |
Yes |
Yes |
|
Llama Nemotron Ultra 253B |
Yes |
Yes |
Model-Based Routing
Model-based routing is static routing. In the request, the user specifies what model the Model Proxy should send the request to. By specifying a target model, Model Proxy can override the model version provided by the user.
Semantic Routing
Semantic routing is dynamic routing where the Model Proxy chooses which model to send the request to. For Sematic Routing, the user creates prompt topics for each route. When a request is sent to the Model Proxy, a semantic service compares the request to the define topic utterances and sends the request to the route that best matches it.
Connections and Model Proxies
Model Proxy includes these types:
-
Proxy: A proxy configuration that can route requests to one or multiple LLM providers by using model-based routing and semantic routing strategies. The enhanced experience manages proxy configurations.
-
Connection: A connection configuration that stores credentials and metadata for connecting to a specific LLM provider. The enhanced experience displays connections as a single-route proxy. Connection configurations are managed by external projects, such as an Agent Network project.