---
title: "Webhooks and events"
description: "Signed events out of Frazer. CloudEvents-shaped JSON for every change, delivered to your endpoints with Standard Webhooks signatures, retries for about 3 days and a 30-day delivery log."
url: https://frazer.ai/docs/webhooks
index: https://frazer.ai/docs/api/llms.txt
tokens: 5442
---

> For agents: this page is indexed at https://frazer.ai/docs/api/llms.txt, and every page at https://frazer.ai/llms.txt. Copy commands from this file exactly.

# Webhooks and events

Frazer sends an event for every change that matters: a server joined, a rollout succeeded, a backup failed, an agent waits for an approval, an invoice was paid. A webhook endpoint receives these events as CloudEvents-shaped JSON, signed with Standard Webhooks and delivered at least once, with retries for about 3 days. Events carry identifiers and states, never secret values.

Product page: [Webhooks and events](https://frazer.ai/integrations#events).

## How it works

-   **Event type**: `ai.frazer.<service>.<resource>.v1.<pastVerb>`: the service (such as `releases`, `kubernetes` or `monitoring`), the resource in it, the version and what happened. For example, `ai.frazer.releases.rollout.v1.succeeded`. Each type has a JSON Schema in the event catalog.
-   **Endpoint**: a URL in a project (`projects/{project}/webhookEndpoints/{endpoint}`) with the event types it receives, a format, its signing secrets and a status: enabled, failing or disabled.
-   **Event sink**: delivery for a whole organization, aggregated over its projects (`organizations/{organization}/eventSinks/{sink}`).
-   **Delivery**: every attempt to send one event to one endpoint, kept in the delivery log with the request, the response and the latency.

Each event has the CloudEvents 1.0 attributes `id`, `source`, `type`, `specversion`, `time` and `subject`, plus `data`. By default `data` is a **snapshot** of the resource at the time of the event. An endpoint can ask for **thin** events instead, which carry only identifiers; you fetch the rest with the API.

```json title="event.json"
{
  "specversion": "1.0",
  "type": "ai.frazer.releases.rollout.v1.succeeded",
  "source": "projects/acme-production",
  "subject": "environments/production/apps/checkout-api",
  "id": "evt_01J9W3…",
  "time": "2026-10-02T10:52:11Z",
  "data": { "revision": "checkout-api-00042", "image": "sha256:4e1c…", "by": "users/noah" }
}
```

```text title="Delivery"
POST https://hooks.acme.example/frazer
webhook-id: msg_2mQ8…
webhook-timestamp: 1759402331
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDDsrqsHU=
```

Slack, Teams and email notifications are rendered from the same events, so a channel and a webhook never disagree about what happened.

## Create an endpoint

1.  Create the endpoint and choose its events. A pattern such as `ai.frazer.releases.rollout.v1.*` matches every rollout event. Match on IDs (`organizationId`, `projectId`), never on display names.
    
    ### CLI
    
    ```sh
    frazer integrations webhook-endpoints create --url https://hooks.acme.example/frazer \
      --events 'ai.frazer.releases.rollout.v1.*' --project acme-production
    ```
    
    ### Console
    
    Open the project’s **Settings › Integrations** and its **Webhooks** tab, add an endpoint, enter the URL and choose the event types and the format. For every project at once, use the organization’s **Settings › Integrations › Webhooks**.
    
2.  Store the endpoint’s signing secret in your receiver. It starts with `whsec_`.
    
3.  Send a test event. In the console, choose **Send test event** on the endpoint, or run:
    
    ```sh
    frazer integrations webhook-endpoints trigger ai.frazer.releases.rollout.v1.succeeded
    ```
    
4.  Check the delivery log. It keeps 30 days of attempts, with the request, the first 4 KB of each response and the latency.
    
    ```sh
    frazer integrations webhook-endpoints deliveries acme-deploy-bot --status failed --since 24h
    ```
    

## Verify signatures

Every request carries three headers. `webhook-signature` holds one or more space-separated signatures of the form `v1,<base64>`: an HMAC-SHA256 of `{webhook-id}.{webhook-timestamp}.{body}`, keyed with your secret (the base64 part after `whsec_`).

```js title="verify.mjs"
import { createHmac, timingSafeEqual } from "node:crypto";

const TOLERANCE_SECONDS = 5 * 60; // your replay window

export function verify(secret, headers, rawBody) {
  const id = headers["webhook-id"];
  const ts = headers["webhook-timestamp"];
  if (Math.abs(Date.now() / 1000 - Number(ts)) > TOLERANCE_SECONDS) return false;

  const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64");
  const expected = createHmac("sha256", key).update(`${id}.${ts}.${rawBody}`).digest();

  return headers["webhook-signature"].split(" ").some((entry) => {
    const [version, signature] = entry.split(",");
    if (version !== "v1") return false;
    const given = Buffer.from(signature, "base64");
    return given.length === expected.length && timingSafeEqual(given, expected);
  });
}
```

-   Sign the **raw** body exactly as received. A body parsed and serialized again does not match.
-   Compare in constant time, and reject old timestamps so a captured request cannot be replayed.
-   Any Standard Webhooks library verifies these requests, and the Frazer SDKs include typed verification helpers.
-   Receivers that prefer public keys can verify Ed25519 signatures, marked `v1a`.

## Handle retries and duplicates

Answer with a 2xx status within 15 seconds; do slow work after you answer. A 3xx counts as a failure. Frazer makes up to 10 attempts, about 3 days in all, and stops at the first success:

| Attempt | After the previous attempt failed |
| --- | --- |
| 1 | Immediately, when the event happens |
| 2 | 5 seconds |
| 3 | 5 minutes |
| 4 | 30 minutes |
| 5 | 2 hours |
| 6 | 5 hours |
| 7 | 10 hours |
| 8 | 10 hours |
| 9 | 24 hours |
| 10 | 24 hours |

-   Delivery is at least once and unordered. Deduplicate on `webhook-id`: a redelivery keeps the same `webhook-id`.
-   When order matters, compare each event’s `time`, or fetch the resource’s current state with the API.
-   To stop the retries for one message, answer with the header `webhook-delivery: abort-message`.
-   After 3 failed attempts, the endpoint’s owner gets an email. After 5 days of failures, the endpoint is disabled, an email goes out, and your other endpoints receive `frazer.webhook.endpoint_disabled.v1`.

## Test a receiver locally

Forward live events to a port on your machine, then send yourself a test event:

```sh
frazer integrations webhook-endpoints listen --forward-to localhost:8080
frazer integrations webhook-endpoints trigger ai.frazer.releases.rollout.v1.succeeded
```

## Redeliver and rotate

Redeliver one delivery from the log with **Redeliver**, or every failure since a time:

```sh
frazer integrations webhook-endpoints redeliver acme-deploy-bot --failed-since 2026-10-02T08:00Z
```

When you rotate an endpoint’s signing secret, the new secret signs alongside the old one for 24 hours, then the old one stops. During those 24 hours each request carries two `v1` signatures: deploy the new secret to your receiver, and the code above accepts either.

## Allow Frazer’s addresses

Frazer sends from fixed egress addresses, IPv4 and IPv6, published with their reverse DNS names in `frazer.ai/egress-ips.json`. Webhooks prefer IPv4. An endpoint can choose IPv6 or either family instead (`source_family`: `PREFER_IPV4`, `IPV6` or `ANY`).

Endpoint URLs are resolved through Frazer’s egress proxy. A URL that resolves to a private, link-local or metadata address is refused, and DNS is checked again when Frazer connects.

## Access and audit

-   Endpoints belong to a project; an event sink delivers an organization’s events from all its projects.
-   Creating, changing, disabling, enabling and deleting endpoints, rotating their secrets and redelivering are audit events: `webhooks.endpoint.create`, `update`, `delete`, `disable`, `enable`, `secret_rotate` and `webhooks.delivery.redeliver`.
-   Events never carry secret values or log bodies. Untrusted text, such as commit messages and issue titles, is shortened and marked as untrusted.
-   An agent that has read untrusted content cannot call a webhook without approval.

## Limits and quotas

| Limit | Default |
| --- | --- |
| Time to answer | 15 seconds, with a 2xx status |
| Retries | 10 attempts over about 3 days |
| Email to the endpoint’s owner | After 3 failed attempts |
| Endpoint disabled | After 5 days of failures |
| Delivery log | 30 days; the first 4 KB of each response |
| Old and new secret both sign | 24 hours |

## Pricing

Webhooks are included in every organization.

| Webhooks and events | On your servers | On the Frazer fleet |
| --- | --- | --- |
| Price | **Included** | **Included** |

Every line, with the estimator: [pricing](https://frazer.ai/pricing#lines).

## Troubleshooting

-   **The signature does not match.** Verify the raw body, not a parsed copy; decode the part of the secret after `whsec_` from base64; check your clock. During a rotation, accept either signature.
-   **An endpoint shows failing.** Your receiver answered with an error or a redirect, or took longer than 15 seconds. Fix it, then redeliver the failures from the log.
-   **An endpoint is disabled.** It failed for 5 days. Fix the receiver, enable the endpoint again and redeliver everything that failed since the first failure.
-   **The same event arrived twice.** Delivery is at least once. Deduplicate on `webhook-id`.
-   **Events arrive out of order.** Delivery is unordered. Compare `time`, or fetch the current state with the API.
-   **The endpoint URL is refused.** It resolves to a private, link-local or metadata address. Use a public address.
-   **Requests never reach a receiver behind an allowlist.** Allow the addresses in `frazer.ai/egress-ips.json`; webhooks use IPv4 unless the endpoint says otherwise.
-   **Events carry only IDs.** The endpoint uses the thin format. Switch it to snapshot, or fetch each resource with the API.

> **Tip:** Keep `frazer integrations webhook-endpoints listen` running while you write a receiver: it forwards real, signed events, so your verification code runs against the same headers it will see in production.

## Event types

Every event type Frazer sends, by product, exactly as it arrives in the `type` attribute. An endpoint subscribes to exact types or to a pattern such as `ai.frazer.releases.rollout.v1.*`.

### [Servers and pools](https://frazer.ai/docs/servers)

| Event type | When it is sent |
| --- | --- |
| `frazer.server.joined.v1` | A server presented a valid join token and appears as pending. |
| `frazer.server.accepted.v1` | A person or a pool rule accepted a pending server. |
| `frazer.server.unhealthy.v1` | A server reported a failing health condition. |
| `frazer.server.burnin.completed.v1` | A server finished its burn-in tests. |
| `frazer.server.removed.v1` | A server was removed from Frazer. |

### [Kubernetes](https://frazer.ai/docs/kubernetes)

| Event type | When it is sent |
| --- | --- |
| `frazer.cluster.created.v1` | A cluster was created. |
| `frazer.cluster.ready.v1` | A cluster reached Ready. |
| `frazer.cluster.upgrade.started.v1` | An upgrade of a cluster started. |
| `frazer.cluster.upgrade.completed.v1` | An upgrade of a cluster completed. |
| `frazer.cluster.degraded.v1` | A cluster's health degraded. |

### [Serverless](https://frazer.ai/docs/serverless)

| Event type | When it is sent |
| --- | --- |
| `frazer.service.deployed.v1` | A deploy of a service finished. |
| `frazer.service.revision.ready.v1` | A new revision is ready and can receive traffic. |
| `frazer.service.revision.failed.v1` | A revision failed to become ready. It receives no traffic. |
| `frazer.stack.created.v1` | A stack and its resources were created. |

### [Devboxes](https://frazer.ai/docs/devboxes)

| Event type | When it is sent |
| --- | --- |
| `frazer.devbox.created.v1` | A devbox was created. |
| `frazer.devbox.stopped.v1` | A devbox stopped (idle, schedule, a person, or a spend cap). |
| `frazer.devbox.change.reported.v1` | The box agent filed a change report for something a person added, removed or changed on a box. |

### [Registry](https://frazer.ai/docs/registry)

| Event type | When it is sent |
| --- | --- |
| `frazer.image.pushed.v1` | A version was pushed to a repository. |
| `frazer.image.scan.completed.v1` | A version's vulnerability scan finished. |

### [Environments and GitOps](https://frazer.ai/docs/gitops)

| Event type | When it is sent |
| --- | --- |
| `frazer.environment.synced.v1` | An environment finished a sync. |
| `frazer.environment.sync_failed.v1` | An environment's sync failed. |
| `ai.frazer.releases.rollout.v1.started` | A rollout started in an environment. |
| `ai.frazer.releases.rollout.v1.succeeded` | A rollout finished successfully. |
| `ai.frazer.releases.rollout.v1.failed` | A rollout failed. |
| `ai.frazer.releases.rollout.v1.rolledback` | A rollout was rolled back. |

### [AI and GPUs](https://frazer.ai/docs/ai)

| Event type | When it is sent |
| --- | --- |
| `frazer.endpoint.ready.v1` | A model endpoint finished deploying and is ready to serve. |
| `frazer.endpoint.failed.v1` | A model endpoint failed to deploy or to stay healthy. |
| `frazer.finetune.completed.v1` | A fine-tuning job finished. |
| `frazer.job.completed.v1` | An AI job ran to completion. |

### [LLM gateway](https://frazer.ai/docs/llm-gateway)

| Event type | When it is sent |
| --- | --- |
| `frazer.llm.budget.threshold.v1` | A gateway budget crossed one of its alert thresholds (50, 80 or 100 %). |

### [Knowledge](https://frazer.ai/docs/knowledge)

| Event type | When it is sent |
| --- | --- |
| `frazer.knowledge.sync.failed.v1` | A source of a knowledge base failed to sync. |

### [Frazer Agents](https://frazer.ai/docs/agents)

| Event type | When it is sent |
| --- | --- |
| `frazer.session.started.v1` | An agent or devbox session started. |
| `frazer.session.waiting_approval.v1` | A session stopped at an action that waits for a person. |
| `frazer.session.completed.v1` | A session finished. |

### [Frazer Skills](https://frazer.ai/docs/agents/skills)

| Event type | When it is sent |
| --- | --- |
| `skills.version.allowed` | A skill version was allowed; created, held, blocked and yanked are sent too. |
| `skills.import.decided` | An import was allowed or refused. |
| `skills.suggestion.opened` | A suggestion to change a skill was opened as a pull request. |
| `skills.gap.opened` | A gap was found where agents struggled with no helpful skill. |
| `skills.lock.refreshed` | A running session moved to a newer skill lock. |

### [Harnesses](https://frazer.ai/docs/agents/harnesses)

| Event type | When it is sent |
| --- | --- |
| `agents.harness.simulation_completed` | A simulation finished, with its results. |
| `agents.harness.assignment_changed` | A use moved to another harness profile. |
| `agents.harness.process_audit_failed` | A harness version failed the process audit and left the uses that need it. |
| `agents.harness.postmortem_published` | A postmortem of a Frazer harness regression was published. |

### [Squads and campaigns](https://frazer.ai/docs/squads/campaigns)

| Event type | When it is sent |
| --- | --- |
| `campaigns.started` | A campaign started, with who started it for whom. |
| `campaigns.tripped` | A campaign's kill switch tripped, with the cause. |
| `campaigns.not_run_no_funds` | Tasks did not start because the budget or the balance could not cover them. |
| `campaigns.successor_opened` | A restarted session's successor opened from its checkpoint. |

### [Frazer Reviewer](https://frazer.ai/docs/reviewer)

| Event type | When it is sent |
| --- | --- |
| `frazer.review.completed.v1` | A review finished and was posted to the pull request. |

### [Lanes, verdicts and outcomes](https://frazer.ai/docs/reviewer/lanes)

| Event type | When it is sent |
| --- | --- |
| `evt.reviewer.verdict.issued` | A verdict was issued for a pull request version. |
| `evt.reviewer.outcome.linked` | An incident, revert or hotfix was linked to the review of the change behind it. |
| `evt.reviewer.miss.analyzed` | A miss analysis finished, with its remedies. |
| `evt.reviewer.lane.revised` | A lane revision entered shadow or was sealed. |

### [Databases](https://frazer.ai/docs/databases)

| Event type | When it is sent |
| --- | --- |
| `frazer.database.created.v1` | A database instance was created. |
| `frazer.database.failover.v1` | A standby took over from the primary. |
| `frazer.database.backup.completed.v1` | A backup finished. |
| `frazer.database.backup.failed.v1` | A backup failed. |
| `frazer.database.restore.completed.v1` | A restore into a new instance finished. |

### [In-memory store](https://frazer.ai/docs/memory)

| Event type | When it is sent |
| --- | --- |
| `frazer.keyvalue.failover.v1` | A replica took over from the primary. |

### [Secrets](https://frazer.ai/docs/secrets)

| Event type | When it is sent |
| --- | --- |
| `frazer.secret.rotate_due.v1` | A secret with the notify rotator is due for rotation. |

### [Frazer Monitoring](https://frazer.ai/docs/monitoring)

| Event type | When it is sent |
| --- | --- |
| `frazer.alert.firing.v1` | An alert policy started firing. |
| `frazer.alert.resolved.v1` | A firing alert policy stopped firing. |
| `frazer.incident.opened.v1` | An alert with a new fingerprint opened an incident. |
| `frazer.incident.acknowledged.v1` | A person acknowledged an incident. |
| `frazer.incident.resolved.v1` | An incident was resolved. |
| `frazer.uptime.check.failed.v1` | An uptime check went down under its failure rule. |
| `frazer.uptime.check.recovered.v1` | A failed uptime check is up again. |

### [Security Center](https://frazer.ai/docs/security-center)

| Event type | When it is sent |
| --- | --- |
| `frazer.finding.opened.v1` | A new finding was opened, or a muted one reopened. |
| `frazer.finding.resolved.v1` | A finding was verified as fixed. |
| `frazer.threat.detected.v1` | A runtime, account, agent or network detection was raised. |

### [Identity and access](https://frazer.ai/docs/iam)

| Event type | When it is sent |
| --- | --- |
| `frazer.member.added.v1` | A person joined the organization or a project. |
| `frazer.member.removed.v1` | A person was removed from the organization or a project. |

### [Billing and budgets](https://frazer.ai/docs/billing)

| Event type | When it is sent |
| --- | --- |
| `frazer.budget.threshold.v1` | A budget crossed one of its thresholds, on actual or forecast cost. |
| `frazer.budget.cap_reached.v1` | A budget with a cap reached 100 % and Frazer paused work in its scope. |
| `frazer.invoice.created.v1` | An invoice was issued. |
| `frazer.invoice.paid.v1` | An invoice was paid. |
| `frazer.invoice.payment_failed.v1` | A payment for an invoice failed. |

### [Integrations](https://frazer.ai/docs/integrations)

| Event type | When it is sent |
| --- | --- |
| `frazer.connection.health_changed.v1` | A connection moved to another health state. |
| `frazer.connection.expiring.v1` | A connection's token, key or certificate expires within 30 days, or its import window ends within 7. |

### [Webhooks and events](https://frazer.ai/docs/webhooks)

| Event type | When it is sent |
| --- | --- |
| `frazer.webhook.endpoint_disabled.v1` | An endpoint was disabled after 5 days of failed deliveries. |
| `frazer.approval.requested.v1` | A change is waiting for an approval. |
| `frazer.approval.decided.v1` | An approval was decided. |
| `frazer.plan.applied.v1` | An approved plan was applied. |
| `frazer.audit.event.v1` | An audit record was written; for a SIEM. |

## Reference

### CLI

| Command | What it does |
| --- | --- |
| `frazer integrations webhook-endpoints create` | Create a webhook endpoint in a project, with the event types it receives. |
| `frazer integrations webhook-endpoints deliveries` | List an endpoint's deliveries, filtered by status and time. |
| `frazer integrations webhook-endpoints redeliver` | Redeliver an endpoint's failed deliveries since a given time. |
| `frazer integrations webhook-endpoints listen` | Forward events to a local port while you develop a receiver. |
| `frazer integrations webhook-endpoints trigger` | Send a test event of a given type. |

All of them: [CLI reference](https://frazer.ai/docs/cli).

### Terraform and OpenTofu

| Resource | What it manages |
| --- | --- |
| `frazer_integrations_webhook_endpoint` | A webhook endpoint; its signing secret is a write-only attribute. |

All of them: [Terraform and OpenTofu reference](https://frazer.ai/docs/terraform).

| Event type | When it is sent |
| --- | --- |
| `frazer.webhook.endpoint_disabled.v1` | An endpoint was disabled after 5 days of failed deliveries. |
| `frazer.approval.requested.v1` | A change is waiting for an approval. |
| `frazer.approval.decided.v1` | An approval was decided. |
| `frazer.plan.applied.v1` | An approved plan was applied. |
| `frazer.audit.event.v1` | An audit record was written; for a SIEM. |

The API is `WebhookEndpointsService` (with deliveries, redeliver, send test and rotate secret) and `EventTypesService`, which serves each event type’s JSON Schema. On `mcp.frazer.ai`, webhook tools are in the toolset `webhooks`.

## Related

-   [Integrations](https://frazer.ai/docs/integrations): the catalog and the connections events can come from.
-   [Frazer Agents](https://frazer.ai/docs/agents): agents can wake on Frazer events and webhooks.
-   [Frazer Monitoring](https://frazer.ai/docs/monitoring): alerts and incidents, and their notification channels.
-   [Identity and access](https://frazer.ai/docs/iam): approvals, plans and the audit log.
-   [Terraform and OpenTofu](https://frazer.ai/docs/terraform): manage endpoints as code.
-   [Webhooks and events](https://frazer.ai/integrations#events): the product page.

## Pages in Connect

- [Integrations](https://frazer.ai/docs/integrations.md)
- [Webhooks and events](https://frazer.ai/docs/webhooks.md) (this page)
- [Terraform and OpenTofu](https://frazer.ai/docs/terraform.md)
- [API, CLI and MCP](https://frazer.ai/docs/api.md)