--- title: "LLM gateway: one safe door to every model · Frazer" description: "One OpenAI- and Anthropic-compatible endpoint for every person, devbox, agent and service, with routes, a key per owner, budgets reserved before each call, guardrails and logs you choose." url: https://frazer.ai/ai/gateway index: https://frazer.ai/llms.txt tokens: 2805 --- > For agents: every page on frazer.ai is indexed at https://frazer.ai/llms.txt. # LLM gateway: one safe door to every model · Frazer **LLM gateway** frazer.ai # One safe door to every model. Budgets that stop. One OpenAI- and Anthropic-compatible endpoint for every person, devbox, agent and service in your organization. Clients ask for a route like balanced; the gateway holds the provider keys, reserves the budget before the call and checks the prompt on the way in. The models on your own GPUs sit behind the same door. [Connect a provider](https://frazer.ai/start?product=llm-gateway) [Read the docs](https://frazer.ai/docs) ``` $ curl -i https://llm.frazer.ai/v1/chat/completions \ -H "Authorization: Bearer $(frazer llm token)" \ -H "content-type: application/json" \ -d '{"model": "balanced", "messages": [ … ]}' HTTP/2 200 x-frazer-request-id: req_01J9W3K7ZP x-frazer-route: balanced x-frazer-route-step: 2 x-frazer-provider: bedrock x-frazer-served-model: claude-sonnet-5-5 x-frazer-credential-source: org-key x-frazer-cost-usd: 0.061 # step 1 · anthropic, your key · 529 overloaded ✓ step 2 · same model, labeled in the reply and the logs ``` ``` { "apiKeyHelper": "frazer llm token", "env": { "ANTHROPIC_BASE_URL": "https://llm.frazer.ai/anthropic" } } ``` ``` # Any OpenAI SDK. OPENAI_API_KEY holds a Frazer key (frz_llm_…) from openai import OpenAI client = OpenAI(base_url="https://llm.frazer.ai/v1") reply = client.chat.completions.create( model="balanced", # a route, not a provider's model name messages=[{"role": "user", "content": "Summarize incident #142"}], ) print(reply.choices[0].message.content) ``` One request from curl, Claude Code or any OpenAI SDK demo organization 01 · How it works Fig. 1 ## Checked before it leaves. Labeled when it comes back. **Fig. 1** The path of one request, and what it leaves behind 02 · Routes and fallbacks Fig. 2 ## Change the model, not the code. Clients ask for a route by name. A route is a stable name with ordered steps, each a provider and a model. Changing the model behind `balanced` is one plan with a revision, and rolling it back is one click. No client changes a line. *The Models and routes tab of the LLM gateway in the demo organization Acme: 7 routes, each a stable name such as balanced, max or coder-small with its primary model, its fallback and its data policy. The balanced route is selected: step 1 is claude-sonnet-5-5 at Anthropic with the organization's key, step 2 the same model through the organization's own cloud account at Amazon Bedrock on zero-retention endpoints; other-model fallback is off and the guardrail policy is prod-default.* **Fig. 2** The routes of a project and the steps of one route product UI, demo organization - ### Same model first When a step fails, the next step serves the same model from another provider, with the data policy checked for every step when you save. - ### Another model only if you say so Set `otherModelFallback` and every such reply carries `x-frazer-served-model` and a degraded mark in the logs. - ### Your key never spends ours A failing key of yours fails its step. Frazer's accounts answer only when the route lists them as a step. - ### Clear triggers A step moves on after a 429, a 5xx, an overload, a lost connection or no first token in time, with one retry per step. ``` # frazer llm routes apply -f balanced.yaml --dry-run name: balanced dataPolicy: zdr guardrailPolicy: prod-default steps: - provider: anthropic # org key, ZDR agreement model: claude-sonnet-5-5 firstTokenTimeout: 20s - provider: bedrock # your cloud account, same model model: claude-sonnet-5-5 otherModelFallback: false perKeyLimits: {rpm: 60, tpm: 400000} cacheMode: provider-only ``` ``` $ frazer llm providers connect anthropic --account frazer --zdr $ frazer llm routes apply -f balanced.yaml --dry-run $ frazer llm keys create acme-billing-worker --routes fast,balanced --budget 200/month --rpm 60 --tpm 200k $ frazer llm token # 1-hour token for the caller $ frazer llm logs --status '!200' --since 1h $ frazer llm usage --by key --month 2026-10 ``` 03 · Guardrails Fig. 3 ## Secrets stop at the door. Checked before any provider or cache. A guardrail policy is a named, versioned set of checks on a route or a key, and a key may only pick a stricter one. Any check can run in monitor mode first, logging what it would have blocked. Your organization's floor is set in Security Center; projects can only tighten it. *The Guardrails tab of the LLM gateway in the demo organization Acme: the policy prod-default with 7 checks in the order they run: secrets in prompts are blocked, personal data is masked and restored in the reply, prompt injection in untrusted text is labeled, secrets in replies are redacted, the data policy and a cost limit of $2.00 per request block, and denied topics are off. A test prompt with an email, a card number and an access key shows the email and card masked and the request blocked before any provider sees it.* **Fig. 3** A guardrail policy and a prompt test product UI, demo organization 04 · What you get ## Built for a whole team. People, boxes, agents and services. - ### OpenAI and Anthropic APIs Chat Completions, Responses and Embeddings, plus Anthropic Messages passed through natively, so prompt caching and thinking survive. - ### Routes with revisions Stable names such as `balanced` or `coder-small`. Change the model in one plan; roll back in one click. - ### A key per owner Services, people, devboxes, agents and knowledge bases. Boxes and agents get 1-hour tokens, never a stored key. - ### Budgets that stop Organization, project, key and session budgets, reserved before each call and settled after. - ### Your GPUs as a provider Every ready Frazer endpoint shows up. Its routes never leave your organization, and boxes share it fairly. - ### Usage you can trace Every reply names its route step, served model and cost; usage rolls up by key, person, box, agent and PR. - ### Guardrail policies Secrets blocked, personal data masked and restored, injection labeled, data policy per route. - ### ZDR mode and a private endpoint Zero data retention per project, and `llm.{project}.internal` inside your project's network. 05 · Controls ## Every key has an owner. And a budget it cannot pass. ### Who may do what Services A key delivered from Frazer Secrets by the cluster’s secret sync. It expires in 90 days; rotation keeps the old secret for 24 hours. People A 1-hour token from `frazer llm token`, or a personal key for scripts that expires within 90 days. Devboxes and agents Never a stored key: a 1-hour token per box, per person on a shared box, and per agent session. Agents Propose key changes over MCP; people apply them. A key never approves anything. Project admins Read stored bodies when a project keeps them, and every read is audited. ### What is logged - A usage row per request: key, route, step, provider, model, credential source, tokens and cost. - **Metadata only** by default, kept 30 days. No prompt or reply bodies. - Masked bodies for 7 days, or full bodies for 7 to 30 days where an organization policy allows it. - A secret found in a prompt is hashed, never stored. - ZDR mode keeps metadata only and denies body capture, the response cache and stateful provider features. ### At the cap - Budgets nest: organization, project, key, session. Periods are a day, a week or a month. - Each call reserves its worst-case cost first. At 100 % a key **blocks**, or only alerts if you choose. - Alerts at 50, 80 and 100 % to the key owner and project admins. - A stream that was admitted is never cut off halfway. - A single request above $2.00 is blocked by default, with the estimate shown. - The organization kill switch stops every key of a kind in one write. 06 · Pricing ## No fee on your own keys. Pay for tokens, not for the door. With your own provider keys the gateway, guardrails, metadata logs and metering cost nothing. Tokens through Frazer's provider accounts appear on your Frazer invoice, inside your project budgets and caps. [See every price](https://frazer.ai/pricing#lines) Gateway Your key: no fee Tokens Frazer-billed providers: list price + 5 % Models on your GPUs No token fee · the GPU's $49 a month covers it Models on the fleet GPU-seconds of the endpoint · [AI and GPUs](https://frazer.ai/ai) Guardrails, logs, metering **Included** Questions ## LLM gateway, answered. What happens when a provider fails? The route moves to its next step after a 429, a 5xx, an overload, a lost connection or no first token in time, with one retry per step. By default the next step serves the same model from another provider. Another model answers only if the route allows it, and the reply says which model it was. A failing key of yours never falls back to Frazer's accounts. Do you keep my prompts? Not by default. The gateway keeps metadata (key, route, model, tokens, cost and guardrail verdicts) for 30 days. Bodies are stored only when a project turns it on: masked for 7 days, or full for 7 to 30 days where your organization allows it, read by project admins with every read audited. ZDR mode turns capture off and allows only zero-retention routes. How do Claude Code and other tools connect? On a devbox, managed settings point `ANTHROPIC_BASE_URL` at the gateway and use `frazer llm token` as the key helper, so the box never holds a stored key. Claude Code always runs on your organization's own Anthropic key, or your own Bedrock, Vertex or Foundry credential, connected to the gateway; Frazer never bills Claude tokens used in Claude Code. Codex, Hermes, the Frazer Harness and OpenCode get the same helper. On a shared box each person gets their own key, so spend and audit stay per person. Can five devboxes share one model on our GPU? Yes. Deploy the model as a Frazer endpoint and publish it to boxes as a shared model; the gateway serves it as a route with a share of tokens per box and per person. When the GPU is busy, interactive requests go first and background ones queue (20 seconds by default), then get a 429 with Retry-After. Which providers can a route use? Anthropic, OpenAI, OpenRouter with your own key, any OpenAI-compatible URL such as Mistral, DeepSeek or Groq, your own Frazer endpoints, and Google Vertex AI, Amazon Bedrock and Azure OpenAI, each on your own cloud account. Routes, keys, budgets, guardrails and logs work the same for every provider. For org-only and air-gapped needs the gateway also runs as a data plane on your own servers, and an exact response cache is opt-in per route. ## More in AI - [**AI and GPUs** Endpoints, tuning, pods](https://frazer.ai/ai) - [**Knowledge** Company RAG, permissioned](https://frazer.ai/knowledge) ## One door to every model. Inside your budgets and rules. [Connect a provider](https://frazer.ai/start?product=llm-gateway) [Read the docs](https://frazer.ai/docs)