LLM Gateway

By default an agent run reads a master provider key straight from its model provider. The LLM Gateway puts a proxy in between. Once enabled, exo deploys a gateway into your connected cluster, syncs your provider keys into it, and hands each run of an agent routed through it a short-lived virtual key instead of the master one. Every request flows through the gateway, which meters token spend, enforces per-agent and per-user budgets, and reports usage back to exo.

It is optional and off by default. A deployment can host multiple gateways, each with its own name, namespace, synced providers, budgets, and access list. Agents opt in individually: an agent that names no gateway keeps using master provider keys directly.

What it does

• Isolates credentials. Your master provider keys are synced into the gateway and only used for proxied traffic. Agents receive a scoped virtual key over which you have full control.
• Meters and caps spend. The gateway prices every request and folds the cost into per-key, per-agent, and per-user counters, enforcing the budgets and rate limits you set.
• Reports usage. Spend, request, and token totals stream back to exo for a per-gateway analytics view.

Create a gateway

Open a deployment's Gateways tab (you must have the gateways:write permission). Two things must already be in place: a connected deployment to host the gateway, and at least one model provider whose keys it will front.

1. Click Create gateway and give it a name. The name doubles as the in-cluster resource name; the namespace defaults to default.
2. exo automatically provisions it on the target operator cluster and syncs in any providers you select.

Routing agents

An agent is routed through a gateway by naming it in its resource's spec.gateway — that reference is the only routing mechanism. When you create an agent in the dashboard, the deployment's default gateway is pre-selected; pick another gateway, or None to keep direct provider keys.

1spec:
2  gateway:
3    name: team-a-gateway
4    namespace: default   # optional; defaults to the agent's namespace

Removing the reference reverts the agent to direct provider keys. The agent's current gateway is shown on its detail page.

Syncing provider keys

The gateway can only front providers whose keys it holds, and syncing is opt-in per provider. On the gateway's Overview tab, each model provider has a Sync action that mirrors its key into the gateway as an upstream credential. Nothing is synced until you choose it, so a provider can exist in exo without ever reaching a gateway.

• Sync pushes the key and, from then on, keeps it in step: editing a synced provider re-syncs it automatically.
• Unsync removes the key and its link from that gateway. Any virtual key pinned to it stops working.
• Deleting a provider removes it from every gateway it was synced to.

Virtual keys

Every request through the gateway carries a virtual key. There are two kinds.

• Session keys are minted automatically for agent runs. They have no fixed expiration date; instead they expire at the end of a session.
• User keys are minted manually from the Virtual Keys tab. A key must name a user and pin exactly one synced provider. You can give it a budget and an expiry, or leave it open-ended.

Any key can be revoked from the Virtual Keys tab; the workload holding it loses model access immediately.

Budgets & rate limits

Rules on the Limits tab bound what runs can spend and how fast they can call. Each rule targets a scope — agent or user — and names one or more targets, or * for all of them. A rule that names a target explicitly overrides the wildcard.

Usage & analytics

The Analytics tab charts spend, requests, or tokens over a 7-, 30-, or 90-day window, filterable by agent and user.

Usage streams back to exo and is stored per gateway, so each gateway has its own history — and it survives even if you remove and re-create the gateway under the same name and namespace.

A sample Gateway

The Gateway can be automatically provisioned through exo or you can manually deploy it on your cluster using the CRD.

1apiVersion: autonomic.sh/v1alpha1
2kind: LLMGateway
3metadata:
4  name: team-a-gateway
5  namespace: default
6spec:
7  replicas: 1
8  resources:
9    requests: { cpu: 200m, memory: 256Mi }
10  service:
11    type: ClusterIP
12    port: 4000
13  # Optional: reference the ConfigMaps holding your rules and prices.
14  budgets: { name: team-a-gateway-budgets, key: rules }
15  pricing: { name: team-a-gateway-pricing, key: pricing }
16---
17apiVersion: v1
18kind: ConfigMap
19metadata:
20  name: team-a-gateway-budgets
21  namespace: default
22data:
23  rules: |
24    - scope: agent          # cap every agent
25      targets: ["*"]
26      total_budget_usd: 100
27      resets: monthly
28      requests_per_minute: 60
29    - scope: agent          # a tighter, specific rule wins over the "*"
30      targets: [billing-bot]
31      session_budget_usd: 5
32      total_budget_usd: 250
33      resets: weekly
34      tokens_per_minute: 100000
35    - scope: user
36      targets: ["*"]
37      total_budget_usd: 50
38      resets: monthly
39---
40apiVersion: v1
41kind: ConfigMap
42metadata:
43  name: team-a-gateway-pricing
44  namespace: default
45data:
46  pricing: |
47    claude-sonnet-5: { input_per_mtok: 3, output_per_mtok: 15 }
48    gpt-4o: { input_per_mtok: 2.5, output_per_mtok: 10 }

Removing a gateway

Remove gateway on the Overview tab tears the instance down: live keys are revoked and the synced upstream credentials are deleted from the cluster. Agents routed through it revert to master provider keys on their next run. You can recreate a gateway at any time.