Contact Us 1-800-596-4880
  1. Homepage
  2. Omni Gateway (1.13)
  3. Securing Omni Gateway Instances with Policies
  4. Inbound Policies Directory

  5. Model Proxy Request Compression

Model Proxy Request Compression Policy

Policy Name

Model Proxy Request Compression Policy

Summary

Compresses message content and replayed tool results in Model Proxy request bodies before they reach the upstream provider, reducing billed tokens

Category

LLM

First Omni Gateway version available

v1.13.0

Release Notes

Model Proxy Request Compression

Returned Status Codes

No return codes exist for this policy.

Summary

The Model Proxy Request Compression policy reduces token consumption by compressing the strings that agents carry in an LLM request body before that body reaches the upstream provider.

In an ongoing agent conversation, each message and each tool result is added to the conversation history. To keep context, a client or runtime might resend the whole history with each new request. Every time it does, the provider charges input tokens for that history again. Compressing those repeated strings reduces the input tokens.

The policy scans the entire request body and rewrites each scope-selected string, keeping the rewritten value only when it is strictly smaller than the original. It supports both the Chat Completions and Responses API shapes.

Two independent compression scopes control which strings are eligible:

  • Compress Tool Results - Compresses every supplied tool result, including replayed history. This covers Chat Completions tool-message content and outputs from Responses function and custom-tool calls. Enabled by default to preserve the policy’s original behavior.

  • Compress Messages - Compresses the Responses instructions field and the textual content of every supplied system, user, and assistant message, including history. Roles, tool calls, metadata, cache breakpoints, and non-text content are preserved. Disabled by default because message content might contain formatting-sensitive instructions or data.

Both scopes traverse the whole request rather than only the latest message. When a runtime resends the raw history in the request body, the policy rewrites it the same way every time, which keeps prompt-cache prefixes stable. History that the provider manages server-side is never part of the request body, so the policy neither transforms nor reconstructs it.

The transforms applied within those scopes are organized into two independent features:

  1. Clean Payload - Compresses each eligible string using a configurable set of transforms: minify (removes only whitespace, so no data is lost), remove null values, remove empty values, and HTML to Markdown. Enabled by default.

  2. TOON Encoding - Re-encodes uniform JSON structures as Token-Oriented Object Notation (TOON), a compact tabular format that reduces tokens for repeated structures. Disabled by default.

The policy is fail-open throughout: any gate miss, parse failure, or result that doesn’t get smaller forwards the original bytes unchanged. When compression does engage, the policy measures the tokens and bytes saved and feeds the token/cost savings to Cost Management telemetry.

Placement

Apply this policy to the inbound Model Proxy chain:

  • After the Model Proxy core processing.

  • Before any provider transcoding policy, so it operates on the request before it is rewritten for a specific provider.

Configuring Policy Parameters

Omni Gateway Local Mode

The Model Proxy Request Compression policy isn’t supported in Local Mode.

Managed Omni Gateway and Omni Gateway Connected Mode

When you apply the policy from the UI, the following parameters are displayed:

The policy configuration is organized into two compression scopes that select which strings are eligible, plus two transform features. Each feature has a switch that enables the fields below it and an Agent IDs allowlist that controls which agents the feature applies to.

Compression Scopes

Element Description Default

Compress Tool Results

Applies the enabled transforms to every supplied tool result, including replayed history. Covers Chat Completions tool-message content and Responses function and custom-tool call outputs.

Enabled

Compress Messages

Applies the enabled transforms to the Responses instructions field and the textual content of every supplied system, user, and assistant message, including history. Roles, tool calls, metadata, cache breakpoints, and non-text content are preserved.

Disabled

Clean Payload

Element Description Default

Enable Clean Payload

Enablement switch for the Clean Payload feature.

Enabled

Minify

Reserializes eligible JSON content in compact form, dropping only insignificant whitespace such as indentation and spaces between tokens. The parsed data is unchanged, so no information is lost.

Enabled

Remove Null Values

Lossy. Recursively drops object keys with null values from eligible JSON content. Null array elements are preserved.

Disabled

Remove Empty Values

Drops object keys whose value is an empty string, array, or object, applied bottom-up until no more can be removed.

Disabled

HTML to Markdown

Converts eligible HTML content to Markdown after stripping <script> and <style> blocks. If the conversion fails, the policy falls back to that stripped HTML instead of erroring.

Disabled

Clean Payload — Agent IDs

Allowlist of agent IDs opted into the Clean Payload feature. For details on allowlist behavior, see Per-Agent Gating.

[] (empty)

TOON Encoding

Element Description Default

Enable TOON Encoding

Enablement switch for the TOON feature.

Disabled

TOON Minimum Response Bytes

Apply TOON encoding only to eligible strings that are at least this many bytes long. A value of 0 disables the threshold.

500

TOON — Agent IDs

Allowlist of agent IDs opted into the TOON feature. For details on allowlist behavior, see Per-Agent Gating.

[] (empty)

Per-Agent Gating

Each feature has its own Agent IDs independent allowlist (Clean Payload — Agent IDs and TOON — Agent IDs). The policy resolves the agent ID from the request and decides to apply the feature to:

  • Direct requests to the Model Proxy that carry no agent ID. The feature always applies to these requests, regardless of the allowlist contents.

  • Requests from an identified agent specified in the allowlist.

You must add an agent to the allowlist for the the feature to apply to its request. There is no allow-all functionality.

In Cost Management, applying the policy to an agent appends the agent’s ID to the allowlist of the enabled feature.

How This Policy Works

For each request, the policy processes the body only when these conditions are met:

  • The request is a POST with a Content-Type of application/json.

  • The request has been normalized to OpenAI shape by the upstream Model Proxy core processing.

When those conditions are met, the policy:

  1. Determines which scopes and features are engaged and if the agent is in a per-agent allowlist.

  2. Scans the full request body and locates the scope-selected strings, in document order, across the complete supplied history.

  3. Rewrites each located string.

  4. Keeps a rewritten string only when it is strictly smaller than the original.

  5. Removes the stale Content-Length header and reserializes the body.

Because the policy is fail-open, an unparseable body, a missing gate, or a result that does not shrink causes the original request to pass through unchanged, and no savings are reported.

TOON Encoding

Token-Oriented Object Notation (TOON) is a compact, JSON-equivalent text format that reduces token consumption for structured data. When Enable TOON Encoding is enabled, the policy re-encodes eligible JSON content as TOON, prefixing the result with `TOON: `.

TOON encoding:

  • Applies only to eligible strings that are at least TOON Minimum Response Bytes.

  • Applies only to the shapes that TOON can shrink, such as uniform arrays of objects and objects with repeated structure.

  • Is lossless but format-changing: the data is preserved, but the model receives it as TOON rather than JSON.

  • Preserves or reduces payload size: the re-encoded value is kept only when it is strictly smaller than the original.

See Also