LLM gateway

The LLM gateway is one OpenAI- and Anthropic-compatible endpoint, https://llm.frazer.ai, for every person, devbox, agent, service and knowledge base in your organization. Clients ask for a route such as balanced; the gateway holds the provider keys, reserves the budget before each call, checks the prompt on the way in and records what the call cost. The models you run on your own GPUs as Frazer model endpoints sit behind the same routes, and their traffic never leaves your organization.

Product overview: LLM gateway.

How it works

One request, end to end:

1. A client sends an OpenAI or Anthropic request to https://llm.frazer.ai, or to the private endpoint llm.{project}.internal inside the project’s network, with a Frazer key: Authorization: Bearer frz_llm_….
2. The gateway checks the key: who owns it, whether it may use the route, the kill switch and its rate limits. It reserves the worst-case cost of the call against every budget above the key.
3. Guardrails check the request: secrets, personal data, injection labels, the data policy and the cost of this one request.
4. The route’s first step calls its provider, with a credential fetched for that request. If the step fails, the next step runs.
5. Output checks run on the reply, streams included, and masked personal data is restored.
6. A usage row (key, route, step, provider, model, credential source, tokens, cost, latency, status) settles the reservation and feeds billing. The reply carries headers that say what served it.

Key concepts

• Provider: a connection to a model provider, or one of your Frazer model endpoints. An admin connects a provider once; a project shares it with other projects of the organization.
• Route: a stable model name, such as balanced, max, fast, coder-small, embed or rerank, with ordered steps. Each step is a provider and a model.
• Key: a Frazer object owned by a service account, a person, a devbox, an agent or a knowledge base. Devboxes and agents never hold a stored key, only tokens that last an hour.
• Budget: a spending limit per organization, project, key and agent session, reserved before each call.
• Guardrail policy: a named, versioned set of checks on routes and keys.
• Logging mode: what the project stores about requests, from metadata only to full bodies, or nothing but metadata in ZDR mode.

Endpoints

Path	What it serves
POST /v1/chat/completions, POST /v1/responses, POST /v1/embeddings, GET /v1/models	The OpenAI API. model is a route name, or provider/model where the project allows direct calls
POST /anthropic/v1/messages, POST /anthropic/v1/messages/count_tokens, GET /anthropic/v1/models	The Anthropic Messages API, passed through natively: cache_control, thinking, tools and anthropic-beta headers arrive unchanged
POST /v1/guardrails:test	Runs a guardrail policy on text without calling a model

Every reply carries these headers:

Header	Value
x-frazer-request-id	The request’s ID, also in the logs and in every error
x-frazer-route, x-frazer-route-step	The route and the step that answered
x-frazer-provider, x-frazer-served-model	The provider and the model that actually served the reply
x-frazer-credential-source	org-key (your key), frazer-account (billed by Frazer) or frazer-endpoint (your GPUs)
x-frazer-cost-usd	The cost of this request
x-frazer-injection-score	The prompt-injection score, when the text was scored

Providers

Connect Anthropic, OpenAI, Google Gemini, Google Vertex AI, Amazon Bedrock, Azure OpenAI, Mistral, DeepSeek, Groq, Together AI, Fireworks AI, xAI, OpenRouter (your own key) or any OpenAI-compatible URL. Frazer also holds its own accounts with providers such as Anthropic and OpenAI, so a route can use them without your key. Google Gemini, Google Vertex AI, Amazon Bedrock and Azure OpenAI connect only through your own cloud account, and Frazer doesn’t bill or resell them. Every ready Frazer model endpoint of kind llm, embedding or rerank in your organization appears as a provider without any setup (AI and GPUs).

Stateful provider features (batch, files, hosted assistants, threads, vector stores, code execution and similar) are off by default in every project and always denied in ZDR mode, because providers do not block them for zero-data-retention accounts.

The gateway runs as a multi-tenant data plane in each regional cell. An organization can also run a dedicated data plane on its own servers for “Org only” and air-gapped needs, and Frazer endpoints on your GPUs are always reached over your own network.

Send your first request

You need a project and the frazer CLI signed in to your organization (CLI reference).

1. Connect a provider.

   CLI

   frazer llm providers connect anthropic --account frazer --zdr

   This connects Anthropic through Frazer’s own account, limited to zero-data-retention use. Frazer bills those tokens.

   Console

   Open AI and GPUs › LLM gateway › Models and routes and choose Providers. Choose Connect on a provider and paste your key; it is stored write-only and never shown again. For each provider, pick your own key or, where Frazer holds one, Frazer’s account, and the model patterns this project may use. Frazer’s accounts are zero-data-retention and no-training by default.

2. Create a route. The fallback step below runs on your own cloud account, connected as a provider. Write the route as a file:

   balanced.yaml

   name: balanced
   dataPolicy: zdr
   guardrailPolicy: prod-default
   steps:
     - provider: anthropic # org key, ZDR agreement
       model: claude-sonnet-5-5
       firstTokenTimeout: 20s
     - provider: bedrock # your cloud account, same model
       model: claude-sonnet-5-5
   otherModelFallback: false
   perKeyLimits: { rpm: 60, tpm: 400000 }
   cacheMode: provider-only

   Then plan and apply it:

   frazer llm routes apply -f balanced.yaml --dry-run

   Output

   Plan: create route balanced in acme-production (revision 1)
     + Step 1        anthropic · claude-sonnet-5-5 · your key · first token within 20s
     + Step 2        bedrock · claude-sonnet-5-5 · your cloud account
     ✓ Data policy   zdr · every step allowed
     ✓ Guardrails    prod-default
       Fallback      same model only
       Per key       60 requests and 400,000 tokens per minute

   Run the command without --dry-run to apply. To change the model behind the route, apply a new revision: it is another plan, and rolling back is one click in the console. Routes, keys and guardrail policies are also resources in the Terraform and OpenTofu provider.

3. Send a request. People get a 1-hour token from the CLI; services use a key.

   curl

   curl -i https://llm.frazer.ai/v1/chat/completions \
     -H "Authorization: Bearer $(frazer llm token)" \
     -H "content-type: application/json" \
     -d '{"model": "balanced", "messages": [{"role": "user", "content": "Summarize incident #142"}]}'

   Python

   # Any OpenAI SDK. OPENAI_API_KEY holds a Frazer key (frz_llm_…)
   from openai import OpenAI
   
   client = OpenAI(base_url="https://llm.frazer.ai/v1")
   reply = client.chat.completions.create(
       model="balanced",  # a route, not a provider's model name
       messages=[{"role": "user", "content": "Summarize incident #142"}],
   )
   print(reply.choices[0].message.content)

   The response headers say what served it. Here step 1 was overloaded, and step 2 served the same model through your cloud account at Amazon Bedrock:

   Response headers

   x-frazer-request-id: req_01J9W3K7ZP
   x-frazer-route: balanced
   x-frazer-route-step: 2
   x-frazer-provider: bedrock
   x-frazer-served-model: claude-sonnet-5-5
   x-frazer-credential-source: org-key
   x-frazer-cost-usd: 0.061

4. Find it in the logs. The Request logs tab shows requests that did not return 200 by default; open one for its route trace and guardrail verdicts. From the CLI:

   frazer llm logs --status '!200' --since 1h

Connect your tools

Point any tool at the gateway’s base URL for its own API format and give it a Frazer key or a token helper. Each path speaks the provider’s native format, so clients change nothing but the URL:

Path	Format
/openai/v1 (or /v1)	OpenAI: chat completions, responses, embeddings, audio, images
/anthropic	Anthropic Messages
/gemini/v1beta	Gemini API, on your own Google account
/bedrock	Bedrock Runtime: invoke and converse, on your own cloud account
/vertex	Vertex AI publisher models, on your own Google project
/mistral/v1, /xai/v1	Those providers’ own extras
/frazer/v1	Frazer’s own: route resolution, guardrail tests, usage

Client	Base URL	Credential
OpenAI SDKs, Codex, OpenCode	https://llm.frazer.ai/openai/v1	A Frazer key, or frazer llm token as the key helper
Anthropic SDKs, Claude Code	https://llm.frazer.ai/anthropic	Same
Hermes and the Frazer agent	Set by the devbox profile	frazer llm token
Services in a cluster	https://llm.frazer.ai/v1, or llm.{project}.internal	A service key delivered to the cluster by its secret sync
Agents	Set in the agent’s sandbox	A token per session from the credential broker

On a devbox, the provisioning profile writes these settings for you. For Claude Code it writes managed settings like these, so the box never holds a stored key and Claude Code fetches a fresh token when the old one expires:

Claude Code managed settings

{
  "apiKeyHelper": "frazer llm token",
  "env": {
    "ANTHROPIC_BASE_URL": "https://llm.frazer.ai/anthropic"
  }
}

Codex, Hermes, the Frazer harness and OpenCode get the same token helper. Claude Code keeps Claude routes, because it only runs Claude models, and it always runs on your organization’s own Anthropic key, or your own Bedrock, Vertex or Foundry credential, connected in Providers: Frazer never bills Claude tokens used in Claude Code. See Devboxes for profiles.

Manage keys, budgets and rate limits

Each key has an owner, the routes it may use, a budget and limits.

Owner	Key	How the client gets it
Service account	Created by hand	Delivered from Frazer Secrets by the cluster’s secret sync. It expires after 90 days; rotation keeps the old secret for 24 hours
Person	Created on first use	frazer llm token (1 hour, refreshed by the CLI), or a personal key for scripts that expires within 90 days
Devbox	One per box, and one per person on a shared box	frazer llm token on the box, at most 1 hour
Agent	One per agent	A token per session from the credential broker, acting for a person, at most 1 hour
Knowledge base	One per base	Used by its ingestion and query services (Knowledge)

Create a key for a service:

frazer llm keys create acme-billing-worker --routes fast,balanced --budget 200/month --rpm 60 --tpm 200k

Budgets nest: organization, then project (the project’s billing budget and its cap), then key, then an optional session budget for agents. Periods are a day, a week or a month, and at 100 % a budget either blocks or only alerts. Alerts go to the key’s owner and the project’s admins at 50, 80 and 100 %.

Reservations stop a call before it passes a budget. When a call is admitted, the gateway reserves its worst case (the estimated input tokens at the input price plus max_tokens at the output price) against every level, and settles it with the real usage when the reply ends. A stream that was admitted is never cut off halfway; the reservation and a per-key concurrency cap keep spend inside the budget.

Rate limits apply per key and per route: requests per minute, tokens per minute (input, output and cached counted separately) and concurrency. A key over its limit gets 429 with Retry-After.

Shared devboxes get one key per person. The box’s token helper asks the box agent who is calling and mints that person’s token, so spend, budgets and audit stay per person. Stopping, reassigning or deleting a box revokes its keys in the same step as its SSH and private-network access.

See what keys spend:

frazer llm usage --by key --month 2026-10

Usage rolls up by key, route, person, devbox, agent, session and pull request.

Configure routes and fallbacks

Clients send a route name, never a provider’s model string, so you can change the model behind balanced without changing a line of client code.

• Triggers: a step moves on after a 429, a 5xx, an overload (529), a lost connection, or no first token within firstTokenTimeout, with one retry per step by default. A context-length error may go to a step with a longer context. A spend cap closes that source until its reset date and is never retried.
• Same model first: by default, the next step serves the same model from another provider. The data policy is checked for every step when you save the route.
• Another model only if you allow it: set otherModelFallback: true. Every such reply carries x-frazer-served-model, its usage row is marked degraded, and the console shows an attention row when more than 1 % of a route’s requests in an hour fall back to another model.
• No fallback: a route may have one step and fail loudly; the console shows a banner during an outage.
• Your key at its limit, Frazer’s capacity next: a key of yours that fails (revoked, unauthorized, a billing error) fails its step. A key that reaches its rate limit or its monthly spend cap moves new work to Frazer’s capacity for the same model, never another, if your organization holds a Frazer budget. Each receipt says so, at list price + 5 % inside that budget. Without a Frazer budget, the work waits with the reason and one action, Allow spillover. Every usage row records its credential source.
• Load balancing: several of your keys in one step are used in turn; Frazer endpoints balance by prefix cache and KV-cache load.
• Data policies: zdr, no-training, org-only or any.

Routes on your own GPUs

Routes on Frazer model endpoints are “Org only”: no step may leave your organization, so instead of falling back, a request waits in a queue (30 seconds by default) and then gets 429.

To share one model with a team’s devboxes, deploy it as a model endpoint, publish it to the boxes as a shared model, and the gateway serves it as a route with a token share per key, which can burst while others are idle. When the endpoint is busy, interactive requests from a box go first, agents next and batch work last; background requests queue (20 seconds by default) and then get 429 with Retry-After. The gateway’s Overview shows how busy the endpoint is, its KV-cache use and time to first token, and offers Add a replica as a plan when it stays above 80 % busy for an hour. An overflow step to an outside model is allowed only after you change the route’s data policy from “Org only”, and every overflowed request is marked degraded.

Restricted models

Some models are restricted by law to certain people or countries. When a rule applies, Frazer keeps the model for organizations that have signed an eligibility attestation and moves every other session to the next model its profile allows, at its next call, with the switch on the receipt. Frazer does not verify each person’s nationality.

Set guardrails

A guardrail policy is a named, versioned set of checks that you attach to routes and keys. A key may only pick a policy stricter than the project’s default. Any check can run in monitor mode first, logging what it would have blocked.

Check	Runs on	Default action
Secrets in prompts	Input	Block, with a typed error to the sender; the value is hashed, never stored
Personal data	Input	Mask with placeholders such as <EMAIL_1>, and restore them in the reply
Prompt injection in untrusted text (tool results, retrieved chunks, web pages, issue bodies)	Input	Label with x-frazer-injection-score; block above 0.9 for agents with write grants
Secrets in replies	Output	Redact, on a sliding window for streams
Data policy	Request	Block if any step may send data to a provider the route’s policy does not allow
Cost per request	Request	Block above a limit ($2.00 by default) and show the estimate
Denied topics, toxicity, grounding	Both	Off by default; an LLM judge, or the guardrail services of your own cloud account (Amazon Bedrock Guardrails, Google Model Armor)

Guardrails run before any cache lookup, so a cached reply never skips them. Try a policy on any text in Guardrails › Test a prompt, or call POST /v1/guardrails:test.

Choose what is logged

Each project picks a logging mode in the gateway’s Settings:

Mode	What is stored	Who can read it
Metadata only (default)	The usage row, route trace and guardrail verdicts, for 30 days; billing aggregates are kept with billing records	Project members, by role
Masked bodies	Bodies after personal data is masked, for 7 days	Project admins; every read is audited
Full bodies	Bodies for 7 to 30 days; needs an organization policy that allows it	Project admins; every read is audited
Time-boxed capture	Masked bodies for one key or route, for at most 24 hours, started from the request drawer	As above
ZDR mode	Metadata only. Body capture, the response cache and stateful provider features are denied, and only zero-data-retention routes run (providers under Frazer’s ZDR agreements, your own ZDR keys, and Frazer endpoints)	Nobody: no bodies exist

Models that require the provider to keep data cannot be used in ZDR mode; the route editor grays them out.

Caching:

• Provider prompt caching passes through untouched (Anthropic cache_control breakpoints and TTLs, OpenAI’s automatic caching). Usage is billed in three classes: input, cache write and cache read.
• An exact response cache is opt-in per route, keyed by key, route and body hash, kept for at most 1 hour in the In-memory store. It is never shared across keys or projects, never used in ZDR mode, and runs after guardrails.
• There is no semantic cache.

Access and audit

• Providers are connected by an organization admin. Provider keys live in your organization’s secret store, are used only by the gateway’s data plane, rotate with overlap and are never shown again. The gateway accepts no provider URLs or credentials in request bodies.
• Routes, keys, guardrail policies and settings are project resources. Every change is a plan with a revision.
• Keys never approve anything. Agents propose key changes with the llm_keys_create MCP tool, and a person applies them.
• Every request leaves a usage row with its credential source, and a span in the project’s tracing.
• Stored bodies are read by project admins only, and every read is audited.
• Kill switch: the organization’s kill switch stops every key of a kind (agents, devboxes, or everything) in one write.

Limits and quotas

Setting	Default
Tokens for devboxes, agents and people	At most 1 hour
Personal keys for scripts	Expire within 90 days
Service keys	Expire after 90 days; the old secret is kept 24 hours after rotation
Retries per route step	1
Cost per request guardrail	$2.00
Metadata logs	30 days
Masked bodies, full bodies	7 days; 7 to 30 days
Time-boxed capture	At most 24 hours
Exact response cache TTL	At most 1 hour
Queue on an “Org only” route	30 seconds, then 429
Queue for background requests on a busy shared model	20 seconds, then 429
Budget alerts	50, 80 and 100 %
Fallback attention row	More than 1 % of a route’s requests in an hour on another model

Pricing

With your own provider keys the gateway costs nothing: guardrails, metadata logs and metering are included. Tokens through Frazer’s provider accounts appear on your Frazer invoice, inside your budgets and caps. Models on your own GPU servers have no token fee, and models on the Frazer fleet bill the endpoint’s GPU-seconds (AI and GPUs). Body capture kept longer than 7 days is billed as log storage.

Tokens Frazer bills cost the provider’s list price plus 5 %; with your own keys, Frazer charges nothing. Frazer publishes the all-in comparison: the cost per completed task through Frazer, through other gateways and directly.

LLM gateway	On your servers	On the Frazer fleet
Price	Your key: no fee	Frazer-billed providers: list price + 5 %
dictation	$0 on your device’s own model	The speech route’s list price per minute of audio + 5 %, billed per second, no minimum

Every line, with the estimator: pricing.

Troubleshooting

Symptom	Cause	Fix
429 budget_exceeded	A budget on the key, project or organization is used up and set to block	frazer llm usage --by key shows who spends; raise the budget with a plan or wait for the next period
429 rate_limited	The key is over its requests, tokens per minute or concurrency on the route	Back off for Retry-After; raise the key’s limits if the load is expected
A reply came from a different model	The route allows another model (otherModelFallback: true) and the first steps failed	Read x-frazer-served-model; turn other-model fallback off if quality matters more than availability
502 provider_unavailable	Every step of the route failed; the error lists the steps tried	Check the provider’s status and your key; add a step with the same model from another provider
400 guardrail_blocked	A check blocked the request; the error names it	Remove the secret, or split a request above the cost limit; test the text in Guardrails › Test a prompt
A model is grayed out in the route editor	The project is in ZDR mode and the model requires the provider to keep data	Pick another model, or use a project that is not in ZDR mode
Requests to a route on your GPUs wait, then get 429	The model endpoint is saturated, and “Org only” routes queue instead of falling back	Add a replica (the gateway offers it as a plan), or allow an overflow step by changing the data policy
Frazer billed tokens you expected on your key	Another step of the route uses Frazer’s account	Check x-frazer-credential-source; if your key was at its cap, that is spillover (see above); otherwise remove the Frazer-account step
Requests moved from your key to Frazer’s capacity	Your key reached its rate limit or monthly spend cap, and your organization holds a Frazer budget	The receipt names the key and the date it reopens; raise the cap at your provider, or turn spillover off in the organization’s settings

Every error carries the request ID. Budget and rate refusals also carry the platform’s quota error details (ErrorInfo in the quota.frazer.ai domain) next to the OpenAI- or Anthropic-shaped body. See Errors.

Reference

CLI

Command	What it does
frazer llm providers connect	Connect a model provider to the project, with your own key or through Frazer's own contract with the provider.
frazer llm routes apply -f	Create or change a route from a YAML file; --dry-run shows the plan first.
frazer llm keys create	Create a key with the routes it may use, a budget and rate limits.
frazer llm token	Print a 1-hour token for the caller, for use as an API key helper.
frazer llm logs	List gateway requests, filtered by status and time.
frazer llm usage	Show usage and cost for a period, grouped by key or another dimension.

All of them: CLI reference.

MCP tools

Tool	What it does
llm_usage_get	Read gateway usage and cost.
llm_routes_list	List the project's routes and their steps.
llm_keys_create	Propose a key change for a person to apply. Writes become a plan that waits for approval.

All of them: MCP tools reference.

Events

Event type	When it is sent
frazer.llm.budget.threshold.v1	A gateway budget crossed one of its alert thresholds (50, 80 or 100 %).

All of them: Events reference.

Errors

Reason	What it means	How to fix it
budget_exceeded	HTTP 429. A budget on the key, the project or the organization is used up for the period, and its action is block.	Raise the budget (a plan) or wait for the next period. Check who spends with frazer llm usage --by key.
rate_limited	HTTP 429 with Retry-After. The key went over its requests per minute, tokens per minute or concurrency on the route.	Retry after the Retry-After time, or raise the key's limits.
guardrail_blocked	HTTP 400, naming the check. A guardrail check blocked the request, for example a secret in the prompt or a request above the cost limit.	Remove what the check found, or test the text against the policy in the Guardrails tab. Policy changes are made by a project admin.
route_not_allowed	HTTP 403. The key may not use this route, or the project does not allow direct provider/model calls.	Use a route the key lists, or add the route to the key.
data_policy_violation	HTTP 403. A step of the route could send data to a provider that the route's data policy does not allow.	Change the route's steps or its data policy.
provider_unavailable	HTTP 502, listing the steps tried. Every step of the route failed.	Check the provider's status and your key, and add a step that serves the same model from another provider.
zdr_endpoint_denied	HTTP 403. The project is in ZDR mode and the request needs something ZDR mode denies, such as a stateful provider feature.	Use a ZDR route and a stateless endpoint, or ask a project admin about the project's mode.

All of them: Errors reference.

Related

• LLM gateway product overview
• AI and GPUs: run models on your GPUs and reach them through the gateway
• Knowledge: company RAG whose model calls go through the gateway
• Devboxes: profiles that set up every tool on a box
• Billing and budgets: project budgets and caps