Contact Us 1-800-596-4880
  1. Homepage
  2. Enhanced MuleSoft Experience
  3. Creating and Managing Model Proxies

  4. Creating Model Proxies

Creating Model Proxies

You can configure a model proxy to route LLM traffic across different models and providers using model-based or semantic routing strategies. When you create a model proxy, you define the endpoint format, routing strategy, and at least one route that maps requests to a supported LLM provider and model. After deployment to Omni Gateway, you can edit the proxy configuration at any time.

A large Omni Gateway supports up to 50 Model Proxies.

Before You Begin

Before getting started, make sure you have:

Create a Model Proxy

  1. In the navigation pane, select Model Proxies.

  2. Click Add Model Proxy

  3. Configure the proxy parameters:

    1. Proxy Name: Define a name for the Model Proxy.

    2. Description: Provide a description of what this proxy does.

    3. Base Path: Define a base path for the proxy endpoint (for example: /billing-ai).

    4. Format: Select an endpoint format:

      • OpenAI: Select the OpenAI API format to send requests to all supported LLM providers (including Gemini and Anthropic). Supports multi-routing and fallback mechanisms. You can’t change this format later.

      • Anthropic: Select the Anthropic API format for native Anthropic Claude model requests. Doesn’t support multi-routing or fallback mechanisms. You can’t change this format later.

      • Gemini: Select the Gemini API format for native Google Gemini model requests. Doesn’t support multi-routing or fallback mechanisms. You can’t change this format later.

    5. Environment: Select the environment for the Model Proxy.

    6. Omni Gateway: Select an Omni Gateway to deploy the Model Proxy to.

    7. Consumer Endpoint: Specify the URL where the Model Proxy will be accessible.

    8. Port: Enter the port number for the Model Proxy (for example, 8081).

    9. Optionally, configure TLS Configuration to secure the proxy endpoint:

      • Secret Group: Select the secret group that contains your TLS context. If you don’t see your secret group, ensure that it is downloadable. For more information, see Add a TLS Context.

      • TLS Context: Select the TLS context to use. If you don’t see your TLS context, click Configure a new TLS Context to create one.

  4. Click Continue.

  5. Configure the routing strategy:

    1. Select a routing strategy:

      • Model-based: Route requests based on the model specified in the request payload.

      • Semantic: Route requests based on semantic analysis of the prompt content. Requires a configured semantic service.

  6. Configure at least one route:

    1. Provide a route name (for example, Route A).

    2. Optionally, click Add headers to add routing headers that filter requests by region, SLA, or custom rules. For example, Region: US

    3. Configure the target:

      • Provider: Select your LLM provider from these options:

        • OpenAI

        • Gemini

        • Azure OpenAI

        • Bedrock Anthropic

        • NVIDIA Nemotron

        • Anthropic

      • Request model override: Select a target model to override the model version specified in the payload. Selecting Use model from request sends the request to the model specified in the request. A target model is required for semantic routing.

      • Destination URL: Specify the URL for your provider endpoint. Ensure the URL is correct and edit if necessary.

    4. Optionally, under TLS Configuration, click Add TLS Context to secure the connection to the provider endpoint with a TLS context.

    5. Configure authentication:

      • API Key: Choose one of the following:

        • Static: Use your provider API key. Enter a static API key for the provider endpoint.

        • Dynamic: Retrieve your key dynamically from the request. Define a DataWeave script to extract the API key from the incoming request.

        • From vault: Use a secret stored in an external vault. Select a secret group, then select a secret from that group. For more information, see Using Credentials Stored in External Vaults.

  7. Click Add Route to add additional routes. Complete the previous steps to configure each new route

  8. Optionally, expand Advanced Options and enable Enable semantic caching to cache and reuse LLM responses based on semantic similarity. Select a configured semantic caching service from the list. For more information, see Configuring Semantic Caching Services.

  9. Click Add Model Proxy.

Edit a Model Proxy

To edit a Model Proxy:

  1. In the navigation pane, select Model Proxies.

  2. Click the name of the Model Proxy you want to edit.

  3. Click Edit Configuration.

  4. Make the necessary edits.

  5. Click Save Changes.