Model Proxy Release Notes

Model proxy now supports semantic caching, which stores and reuses LLM responses based on semantic similarity to reduce latency and cost. Semantic caching supports Azure AI Search as the vector database.

You can now secure model proxy connections with TLS by configuring a TLS context on the proxy consumer endpoint and on connections to provider endpoints.

Model proxy can now authenticate to LLM providers by using a secret stored in an external vault (AWS Secrets Manager, Microsoft Azure Key Vault, or HashiCorp Vault).

The Model Proxy Request Compression policy is now available for model proxies.

LLM Proxy is now renamed to model proxy.

Model proxy now supports the Anthropic API format and native Anthropic models.

Model proxy now supports Advanced Scale semantic services that use vector databases to store up to 2000 utterance vectors per prompt topic.

Agent Network now supports model proxies as Agent Broker LLM providers.

Model proxy can now dynamically extract API Keys (such as OAuth 2.0 tokens) from incoming requests to support dynamic LLM provider authentication.

Model proxies now support NVIDIA Nemotron models.

Anypoint Monitoring now provides new dashboards for cost management.

Known Issues