---
title: "Deploy an open model"
description: "gpt-oss or Llama on your GPUs or the fleet."
url: https://frazer.ai/docs/quickstarts/open-model
index: https://frazer.ai/docs/llms.txt
tokens: 1730
---

> For agents: this page is indexed at https://frazer.ai/docs/llms.txt, and every page at https://frazer.ai/llms.txt. Copy commands from this file exactly.

# Deploy an open model

At the end of this guide, gpt-oss-120b runs as a **model endpoint** in the project `acme-production`. Frazer runs its inference engine behind an OpenAI-compatible API, adds workers as requests queue up and scales to zero when the endpoint is idle. You try it in the Playground and call it from code.

You need:

-   A project. This guide uses `acme-production`.
-   GPUs: GPU servers of yours in a pool, or the Frazer fleet. A GPU server joins like any other server; its GPUs are found during inspection.
-   GPU quota for the class in the project. An endpoint scaled to zero holds none.
-   For a gated model such as Llama: a Hugging Face account with access to it.

## Deploy the model

1.  **Install the CLI and sign in.**
    
    ```sh
    curl -fsSL https://get.frazer.ai | sh
    frazer login
    ```
    
2.  **For a gated model, connect Hugging Face.** gpt-oss is not gated; skip this step for it. For Llama, open **AI and GPUs › Integrations** and connect Hugging Face with OAuth or a fine-grained token. Frazer downloads gated weights with your own credential, keeps them in your organization’s storage and checks your access again on every new pull.
    
3.  **See where GPUs are free.** In the console, open **AI and GPUs › Overview** and select the **Capacity** tab. For each GPU class it shows the GPUs free and in use on your own servers, the fleet’s free GPUs in each region (live), your project’s quota and the jobs queued. The plan in the next step shows the same for the class you pick.
    
4.  **Plan the deploy.**
    
    ```sh
    frazer ai endpoints deploy hf://openai/gpt-oss-120b --project acme-production \
      --gpu h100-80gb --min 0 --max 3 --dry-run
    ```
    
    ```text title="Output"
    Plan  projects/acme-production/endpoints/gpt-oss-120b
      + Import     hf://openai/gpt-oss-120b · MXFP4
      + Endpoint   gpt-oss-120b · Frazer inference · 1× H100 80GB · min 0, max 3
      Where        fleet us-virginia1 · your pool sf-office-default has no h100-80gb free
      Cost         per GPU-second while a worker is ready · nothing at zero
    ```
    
    The plan lists every place the endpoint fits, your own GPUs first, each with its price and live free count. The import pins the model to an immutable revision and downloads only the files its format needs. The model’s license is shown, and you accept it with the import.
    
    To choose the place yourself, or to keep the endpoint in Git, describe it in a file and plan it with `frazer apply -f gpt-oss-120b.yaml --dry-run`:
    
    ```yaml title="gpt-oss-120b.yaml"
    apiVersion: ai.frazer.ai/v1
    kind: Endpoint
    name: projects/acme-production/endpoints/gpt-oss-120b
    spec:
      kind: llm
      model: hf://openai/gpt-oss-120b
      gpu: { class: h100-80gb, count: 1 }
      placement: { location: us-virginia1 } # or { pool: sf-office-default }
      scaling: { min: 0, max: 3 }
    ```
    
5.  **Deploy.** Run the same command without `--dry-run`:
    
    ```sh
    frazer ai endpoints deploy hf://openai/gpt-oss-120b --project acme-production \
      --gpu h100-80gb --min 0 --max 3
    ```
    
    ```text title="Output"
    ✓ Imported      hf://openai/gpt-oss-120b
    ✓ Weights cached
    ✓ Ready
    → https://gpt-oss-120b-4f7a2c.frazer.run/v1
      OpenAI-compatible · key in the LLM gateway
    ```
    
    The endpoint’s page, under **AI and GPUs › Inference › Model endpoints**, shows its state, workers, engine and version, quantization, maximum context, and cold-start times.
    
6.  **Try it in the Playground.** On the endpoint’s page, select the **Playground** tab and send a prompt; each answer comes with its metrics. **AI and GPUs › Inference › Playground** compares up to three models side by side, and **View code** shows the request it sent.
    
7.  **Call it from code.** Any OpenAI client works; this one is the Python SDK (`pip install openai`). Point it at the endpoint’s URL, which **Call it** on the endpoint’s page shows, and authenticate with a token from the LLM gateway; `frazer llm token` prints a 1-hour token for you:
    
    ```python title="hello.py"
    import subprocess
    from openai import OpenAI
    
    
    token = subprocess.run(["frazer", "llm", "token"], capture_output=True, text=True, check=True).stdout.strip()
    client = OpenAI(base_url="https://gpt-oss-120b-4f7a2c.frazer.run/v1", api_key=token)
    
    
    model = client.models.list().data[0].id
    reply = client.chat.completions.create(model=model, messages=[{"role": "user", "content": "Say hello."}])
    print(reply.choices[0].message.content)
    ```
    
    A service that calls the model gets its own key (`frazer ai endpoints keys create`), with a budget and rate limits.
    

## Serve Llama instead

Connect Hugging Face (step 2), then deploy the model by its Hugging Face ID, for example `hf://meta-llama/Llama-3.3-70B-Instruct`, with a GPU class it fits on. The **Model catalog** under **AI and GPUs › Inference** shows which classes each model fits on, and accepts a pasted Hugging Face URL to start an import.

## Scaling and cost

-   **Scale to zero.** With `--min 0`, workers stop after 10 minutes without requests. The next request is held while a worker starts; after 5 minutes without one it gets a 503 with `Retry-After`. For large models, keep one worker warm with `--min 1`.
-   **On the Frazer fleet** you pay per GPU-second by class, from the moment a worker is ready. Starting, throttled, unhealthy and stopped workers are not billed.
-   **On your own GPU servers** a flat fee per GPU covers endpoints, pods and notebooks; usage shows as GPU-seconds so each team sees its share.

Prices for both are on the [pricing page](https://frazer.ai/pricing#lines).

> **Note:** Each organization on the fleet gets its own virtual machine with whole GPUs. On your own servers, only your organization’s work runs.

## Next steps

[AI and GPUs](https://frazer.ai/docs/ai) Endpoints, serverless endpoints, GPU pods, fine-tuning and quotas.

[LLM gateway](https://frazer.ai/docs/llm-gateway) Keys, routes, budgets and logs for every model you call.

[Bring your first server](https://frazer.ai/docs/quickstarts/first-server) Join a GPU server of your own.

[AI and GPUs on frazer.ai](https://frazer.ai/ai) The product page, with GPU classes and fleet prices.

## Pages in Get started

- [Docs home](https://frazer.ai/docs.md)
- [Bring your first server](https://frazer.ai/docs/quickstarts/first-server.md)
- [Create a Kubernetes cluster](https://frazer.ai/docs/quickstarts/kubernetes-cluster.md)
- [Deploy serverless with Postgres](https://frazer.ai/docs/quickstarts/serverless-postgres.md)
- [Deploy an open model](https://frazer.ai/docs/quickstarts/open-model.md) (this page)
- [Give your team devboxes](https://frazer.ai/docs/quickstarts/devboxes.md)
- [Connect your environment repo](https://frazer.ai/docs/quickstarts/environment-repo.md)