---
title: "Frazer Monitoring"
description: "Metrics, logs and dashboards for every project, uptime checks from three locations, alert policies you backtest before they page, and incidents that open with a read-only triage."
url: https://frazer.ai/docs/monitoring
index: https://frazer.ai/docs/llms.txt
tokens: 6520
---

> For agents: this page is indexed at https://frazer.ai/docs/llms.txt, and every page at https://frazer.ai/llms.txt. Copy commands from this file exactly.

# Frazer Monitoring

Frazer Monitoring collects metrics and logs from every cluster in a project from the moment it is created, and gives every project its own dashboards, already built. The storage runs in a monitoring cell that Frazer operates in your region, whether your workloads run on your servers, in your cloud accounts or on the Frazer fleet. On top of that data, uptime checks probe your endpoints from three locations, alert policies page the people you route them to, and every incident opens with a read-only triage that names the last change.

Product overview: [Frazer Monitoring](https://frazer.ai/monitoring).

## How it works

Every cluster starts with Frazer Monitoring’s metrics agent and log collector. They ship metrics and logs outbound only to the monitoring cell in your Frazer region; for clusters on your own servers the data travels through Frazer Link. There is no Prometheus server to run: the operator reads `ServiceMonitor`, `PodMonitor` and `PrometheusRule` resources, so third-party Helm charts work unchanged. Anything else you want to collect, such as your own exporters or another log shipper, you add through Frazer GitOps like the rest of your apps.

Each project is its own tenant in the cell. Every read, from the console, from dashboards, from the CLI or from a query the assistant writes, goes through a proxy that verifies your Frazer token and pins the tenant, so a query reads one project’s data and nothing else. The log pipeline drops known token formats and anything tagged secret before it is stored.

Alerts come from three places and go through one notifier, which deduplicates them, groups them into incidents and sends them to your channels.

-   **Tenant**: one project’s metrics, logs and traces. Agents attach the tenant themselves; a workload cannot choose or change it.
-   **Dashboard**: a view definition, a record of the project that the console and the Frazer app draw over the query APIs, with an **All projects** view for organization admins. Every panel’s query runs as the viewer, so a dashboard never shows what its viewer could not query.
-   **Uptime check**: a probe of an endpoint (HTTP, TCP, TLS, DNS, ICMP, gRPC) or a heartbeat your job sends, run from several locations.
-   **Location**: where a check runs. Public locations are Frazer’s, on different providers; private probes run inside your network.
-   **Alert policy**: a condition on a check, a PromQL query or a log query, with a severity, a runbook and a route.
-   **Incident**: one per alert fingerprint, with a timeline, an owner and a postmortem.
-   **Channel**: where notifications go: email, Slack, Microsoft Teams, Discord, a signed webhook or PagerDuty.
-   **Silence** and **maintenance window**: rules that hold alerts for a while, with a reason and an author.

## Get started: add an uptime check

An uptime check is the fastest way to see Frazer Monitoring end to end: a check, a policy on it, a channel, and an incident when it fails.

1.  In the console, open **Project › Monitoring › Uptime checks** and choose **Create uptime check**.
    
2.  Choose the type and the target. For an HTTPS endpoint, set the method, the expected status, and optionally the body text or JSON path to look for and the maximum latency. Conditions use a small expression language:
    
    ```text title="Conditions"
    [STATUS] == 200
    [CERTIFICATE_EXPIRATION] > 48h
    ```
    
    Header values such as tokens are stored in the secrets store and added at the probe. Nobody sees them again, people or AI tools.
    
3.  Choose the locations and the interval. Keep the default failure rule: the check is down when at least 2 locations fail in the same interval. Keep **IP family** on `BOTH` so IPv4 and IPv6 are checked separately.
    
4.  Choose **Test now** and pick the locations to run it from. Each location reports its status, timings and any error before you save.
    
5.  Save the check. Results land in the project’s own metrics (`probe_success`, `probe_duration_seconds`, `probe_http_status_code`, `probe_ssl_earliest_cert_expiry`, `probe_dns_lookup_time_seconds`, labeled by check, location and IP family) and one log line per run.
    
6.  Open **Monitoring › Alerts › Policies**, create a policy with the source **Check**, pick the check, and route it to a channel. If you have no channel yet, add one under **Project › Settings › Alerting** and use **Send test** before you rely on it.
    

The check’s page shows uptime bars, latency per location, the run log, the TLS certificate, and the policies and incidents that use it.

## Choose a check type

| Check | What it checks |
| --- | --- |
| HTTP(S) | Method, headers, body, expected status, body text or regular expression, JSON path, maximum latency, up to 5 redirects |
| TCP | Connect, with optional TLS |
| TLS expiry | Days left, chain validity, hostname match |
| Heartbeat | A ping URL your cron job calls, with start and fail signals; a period and grace, or a cron expression with a time zone |
| DNS | Record type, expected values, resolver |
| ICMP | Ping, with a packet-loss threshold |
| gRPC health | The standard `grpc.health.v1` service |

**IPv4 and IPv6.** HTTP(S), TCP, TLS, ICMP and gRPC checks have an IP family. `BOTH`, the default, resolves A and AAAA and probes each family on its own, so a broken AAAA record shows as **IPv6 down**, not **Up**. When the target has both records, the check is down if either family fails. `IPV4` and `IPV6` probe one family; `ANY` connects the way a browser does. A target with only an A record is checked over IPv4, and the check says so.

**The failure rule.** Each location runs the check once per interval and retries once, immediately, before it counts as failed. The check is down when at least `min_failing_locations` (default 2, of 3 or more) fail in the same interval. With `BOTH`, the rule runs per family. Every Frazer location also checks known-good targets; when a location’s own checks fail, its results are set aside and Frazer’s staff are paged instead of you.

**Allow-listing the probes.** Public probes send from their own published addresses, two IPv4 and two IPv6 per location, listed at `frazer.ai/probe-ips.json`, with the User-Agent `FrazerProbe/1.0 (+https://frazer.ai/probes)`.

### Monitor a cron job with a heartbeat

Create a **Heartbeat** check with the job’s period and a grace time, or its cron expression and time zone. The check gives you a ping URL with a token; call it from the job.

```sh title="backup.sh"
curl -fsS https://hb.frazer.ai/<token>/start
pg_dump acme > /backups/acme.sql && curl -fsS https://hb.frazer.ai/<token> \
  || curl -fsS https://hb.frazer.ai/<token>/fail
```

The check fails when no ping arrives within the period plus the grace, or when the job reports a failure.

### Check private endpoints

Public probes refuse private, loopback, link-local and other reserved addresses, and every Frazer range. To check a Service inside a cluster or an endpoint on your network, use a private probe:

-   **In your Frazer clusters**, the probe is a module of `frazer-clusterlet`, which already runs in every cluster; choose the cluster as a location.
-   **Anywhere else**, run `frazer-probe` as a container or a binary. Get its install command and probe token from **Project › Settings › Alerting › Private probes**. It keeps one outbound gRPC stream to Frazer, receives its checks and sends results; it opens no inbound port.

A private probe belongs to one project and runs that project’s checks, plus checks of projects you share it with from its **Share** tab.

## Create an alert policy

A policy watches a check, a metrics query or a log query, and you test it on your own data before it can page anyone. Frazer turns policies into alerting rules for you; you never write rule files by hand.

### Console

1.  Open **Project › Monitoring › Alerts › Policies** and choose **New policy**.
    
2.  Pick the source: **Check** (its up or down state), **Metrics** (PromQL with a threshold and a duration) or **Logs** (a log query’s count or rate).
    
3.  Write the query, or describe it in words, for example _Page us when more than 5% of checkout-api requests fail for 5 minutes_. The assistant drafts the query; you see it, edit it and run it read-only before anything is saved:
    
    ```text title="PromQL"
    sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5..", service="checkout-api"}[5m]))
      / sum(rate(http_server_request_duration_seconds_count{service="checkout-api"}[5m]))
    ```
    
4.  Set the threshold (here `> 0.05`), **For** (5 minutes), the severity, labels, annotations and a runbook URL.
    
5.  Read the **Backtest**: seven days of your data show how often the policy would have fired, and for how long.
    
6.  Set the route: channels, grouping, repeat interval, and whether to notify on resolve. Save.
    

### Rules in Git

Keep `PrometheusRule` resources in your environment repository. Frazer GitOps applies them like the rest of the app, and the same notifier routes what they fire.

```yaml title="envs/production/alerts/acme.yaml"
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: acme
spec:
  groups:
    - name: notifications
      rules:
        - alert: Notifications5xxHigh
          expr: |
            sum(rate(http_server_request_duration_seconds_count{
              http_response_status_code=~"5..", service="notifications"}[5m]))
            / sum(rate(http_server_request_duration_seconds_count{
              service="notifications"}[5m])) > 0.01
          for: 5m
          labels:
            severity: critical
```

Every cluster also gets Frazer’s default rules: node and disk pressure, certificate expiry, backup age, GitOps out of sync, and pods in a crash loop. You can mute them per project; you cannot delete them.

## Route alerts and run incidents

**Channels** are project resources under **Project › Settings › Alerting › Channels**: email, Slack, Microsoft Teams, Discord, webhooks signed with Standard Webhooks, PagerDuty, and Opsgenie until its shutdown in April 2027. Each channel is verified before use (an email confirm link, Slack OAuth) and has a **Send test** button and a rate limit. Share a channel with other projects of the organization from its **Share** tab.

**Routing** sends alerts by label, severity and project, with grouping and a repeat interval per route. For example: everything to `#acme-alerts`, critical only to PagerDuty, critical and resolve notices to `oncall@acme.example`. On-call schedules and escalation stay in PagerDuty, which Frazer bridges to.

**Incidents** open from an alert fingerprint, so a repeating alert updates one incident instead of opening new ones. An incident moves from **Triggered** to **Acknowledged** to **Resolved**. Its timeline records alerts, notifications, acknowledgments, comments, AI notes and status changes, each with who did it, and you write the postmortem in Markdown on the incident page.

**AI triage.** When an incident opens, the assistant runs a read-only investigation, stopped after 60 seconds and a fixed number of tool calls. It posts to the timeline and to Slack: what fired, the blast radius (locations, checks, workloads), the changes of the last 2 hours from the audit log, unusual log patterns, and runbook steps with links to the evidence. It changes nothing by itself: a fix it suggests is a plan a person approves. Log lines and HTTP bodies are treated as untrusted input.

**Silences** hold alerts that match label matchers, with a start, an end, a reason and an author. **Maintenance windows** are one-off or recurring (an RRULE), per check or label set; they suppress alerts and can show on a status page. Server maintenance on a shared pool appears in each project’s calendar.

## Query logs and metrics from the terminal

`frazer logs` and `frazer monitoring metrics` read the monitoring cell, so they work without access to the cluster.

```sh
frazer logs --cluster acme-production-1 \
  'namespace:acme level:(error OR warn)' --since 15m
```

```text title="Output"
14:02:11  error  notifications  upstream timeout calling orders-db pooler
14:02:11  error  checkout-api   POST /notifications.v1.Messages/Send → 503
14:02:10  warn   notifications  pooler: client waiting 1.4 s
14:02:10  warn   checkout-api   circuit half-open for notifications
```

In the console, **Monitoring › Logs** adds facets, patterns, surrounding context, live tail and saved queries, and turns a query into an alert policy. **Monitoring › Metrics** is the metrics explorer for PromQL.

## Build and share dashboards

**Built-in dashboards** appear in every project under the folder **Frazer/** as you use a product. They are view definitions that update with Frazer and are read-only; **Save as copy** makes one yours. A built-in that starts from an open-source project’s dashboard is converted when Frazer builds it, and one that does not convert cleanly is redrawn before it ships, never shipped with a missing panel.

| Dashboard | Appears when |
| --- | --- |
| Project overview; Uptime checks; Alerts and incidents; Cost and usage | Always |
| Kubernetes cluster, Namespace, Workload, Node | A cluster exists |
| Serverless service, Job | A service exists |
| Managed Postgres | A database instance exists |
| In-memory | An in-memory instance exists |
| Gateways and routes | A gateway exists |
| LLM gateway usage | The LLM gateway is in use |
| GPU | GPU pools exist |
| Devboxes, Agent sessions | Those products are in use |

Every built-in dashboard uses the same variable names (`project`, `cluster`, `service`, `revision`, `instance`), so **Open dashboard** on a resource page lands with the variables set.

**Your own dashboards** are made in the console’s dashboard editor, or by an agent through the API, the CLI and MCP. Folders carry View, Edit and Admin permissions that cascade, granted to your Frazer groups. Deploys, rollouts, incidents, maintenance windows and the fix agent’s merges are drawn on every time series. **Monitoring › Dashboards** lists built-in, custom, repository and imported dashboards, with folders, owners and stars. Creating a dashboard or saving a copy is an audited action.

**Templates.** The gallery holds Frazer dashboards that are not installed by default (SLO burn rate, Go runtime, GitOps) and imports dashboard JSON from the tools you used before, by grafana.com ID or as a file. Frazer fetches imports through the same guard that protects the probes, converts each into a view definition with a report per panel, and checks each variable’s query against your project. An import is a starting point: what you keep is Frazer’s view definition.

**Sharing.** A console link carries the dashboard, its variables and the time range. Snapshots freeze the data, redacted by the project’s rules. Public (externally shared) dashboards are off by default; an organization Owner can turn them on, and a public dashboard is served from a pre-rendered copy, never from the live queries.

### Keep dashboards as code

#### Repository

Keep view definitions (`dashboards/*.yaml`, kind `Dashboard` in `monitoring.frazer.ai/v1`) in a repository folder, such as `dashboards/` in `acme/platform-envs`, synced both ways with verified commits: a save in the console becomes a commit, or a pull request where the branch is protected. A GitHub repository connects through the **Frazer Reader** GitHub App, which reads; to save changes back as pull requests, install the **Frazer** app, which writes.

#### CLI

```sh
frazer monitoring dashboards apply -f dash.yaml --dry-run
```

The CLI writes the dashboard through the dashboards API with an audit entry; `--dry-run` shows the diff and the queries’ estimated cost first.

#### Dashboards API

`DashboardService` in `frazer.monitoring.v1` (`Get`, `List`, `Create`, `Update`, `Delete`, `Render`, `Export`) and the MCP `dashboards_*` tools. Access is IAM: `monitoring.dashboards.get`, `.list`, `.create`, `.update` and `.delete` on the project, inherited by folders. Exports are Frazer’s own YAML or JSON, so no dashboard is locked in.

## Turn on traces

Traces are opt-in per cluster. Turning them on adds an OpenTelemetry collector that sends metrics, logs and traces to the monitoring cell, with auto-instrumentation for Java, Node.js, Python and .NET. Go is not instrumented by default, because it needs a privileged sidecar. Before you turn traces on, choose the allow-list of resource attributes that become labels: promoting pod IDs makes the series count explode. Traces are kept 3 to 7 days. Metric-to-trace links are queries; metrics carry no exemplars.

## Publish a status page

A status page is a project resource under **Project › Settings › Alerting › Status pages**. Its components map to checks or groups of checks, its uptime bars come from your metrics, and its incident and maintenance posts link to your incidents and windows. Visitors subscribe by email, RSS or webhook. Give it a custom domain with an ACME certificate, and make it public, password-protected or limited to Frazer SSO. Pages are rendered static and served from a CDN, so they stay up when the Frazer control plane does not. A page can show components of checks shared to it from other projects of your organization.

## Agents that file and fix issues

The monitoring agent, `monitor`, turns what your services do into [issues](https://frazer.ai/docs/source/issues) in Fir: new and regressed error groups, alert firings at `ticket` severity, slow error-budget burns, CI failure clusters, canary failures and dependency advisories, and, in shadow until their precision is measured, new log patterns, metric anomalies and UI friction. Filing is deterministic: windows by impact tier, fingerprints that update an open issue instead of filing a second, an owner from `OWNERS`, a severity from a table and a confidence from the rule’s measured precision. Budgets per team and service fold the overflow into one digest issue a day, and during an incident new symptoms attach to the incident instead. Each issue arrives with its evidence (the sampled events, traces, the log window, the release’s changes), redacted before storage and re-runnable. The fix agent, `fixer`, then reproduces, writes the test first, proves and merges the fix under your issue policy, and verifies it in production against these same signals.

**On by default with monitoring.** When a project turns the monitoring agent on, the fix agent runs too, for risk classes R0 to R2 through its nine gates and a person for R3, with the expected spend shown first and the organization’s monthly cap, budgets and kill switches bounding it. **Values from production** (read-only probes and exception snapshots) need a **capture approval** a person gives per service, bounded, redacted and failing closed; previews allow probes by default ([Issues](https://frazer.ai/docs/source/issues#values-from-production-probes-and-snapshots)). `frazer monitoring capture-approvals list` shows every capture approval on your services, payload capture for [Frazer QA](https://frazer.ai/docs/previews/tests#production-payloads-only-on-an-opt-in-per-service) included.

## Access and audit

Dashboards use your Frazer project roles; there are no separate dashboard roles.

| Role | In Monitoring |
| --- | --- |
| Viewer, Audit viewer | Read metrics, logs, dashboards and incidents |
| Responder | Also acknowledge and resolve incidents, and set silences and maintenance windows; change no resources |
| Developer | Create checks, policies, channels and dashboards |
| Admin, Owner | Everything above; only an Owner turns on public dashboards |

**Agents** read through the `monitoring`, `logs`, `metrics` and `incidents` MCP toolsets. A change they propose is a plan a person approves; see [Identity and access](https://frazer.ai/docs/iam#approve-changes).

**What is logged.** Every create, change and delete of a check, policy, channel, silence or status page is an API call that accepts `validate_only` and `request_id` and leaves an audit entry. A policy records who created it and whether an agent drafted it; a silence records its author and reason; an incident keeps its full timeline.

> **Note:** Dashboards carry no alerting of their own. Every alert comes from Frazer’s policies, defaults and rules in Git, so there is one place to silence and route them.

## Limits and quotas

| Item | Default |
| --- | --- |
| Metrics retention | 30 days |
| Logs retention | 30 days |
| Traces retention | 3 to 7 days |
| Uptime check interval | 3 to 5 minutes on the free tier, 60 seconds on paid plans, 30 or 10 seconds on the high tier for verified organizations |
| Check timeout | 30 seconds at most |
| Retries before a location counts as failed | 1, from the same location |
| Failure rule | 2 failing locations of 3 or more |
| HTTP redirects followed | 5 |
| Response body | Read up to 1 MiB; kept in the run log up to 10 KB |
| AI triage | 60 seconds and a fixed number of tool calls per incident |
| Included per server you bring | 50,000 active metric series, 50 GB of logs a month, 20 uptime checks; pooled across the billing account |

Each project also has ingest and cardinality limits; the agent drops high-cardinality labels before they are stored. Free plans probe only an allowed list of TCP ports, and every destination host has a request cap across all customers, so probes cannot be used as a scanner.

## Pricing

Every server you bring adds to an allowance of metric series, logs and uptime checks, pooled across the billing account; above it you pay per GB of logs and per 1,000 active series. When a budget with a cap reaches it, monitoring above the allowance is sampled down to the allowance, and alerts are never dropped.

| Frazer Monitoring | On your servers | On the Frazer fleet |
| --- | --- | --- |
| Price | 50,000 series, 50 GB logs (kept 30 days) and 20 uptime checks per server included | Logs $0.20 / GB · metrics $1 per 1,000 series · checks $1 per check / month |

Every line, with the estimator: [pricing](https://frazer.ai/pricing#lines).

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| A check shows **Degraded** but nobody was paged | Only 1 of 3 locations failed, below the failure rule | Look at that location’s run log. Change **min failing locations** only if one location should be enough |
| A check is down with **IPv6 down** while the site loads in your browser | The AAAA record or the IPv6 path is broken; browsers fall back to IPv4 | Fix the AAAA record or the IPv6 route, or set the check’s IP family to `IPV4` if the service has no IPv6 |
| An HTTP check to an internal address is refused | Public probes refuse private, reserved and Frazer addresses | Run the check from a private probe in that network |
| Your firewall blocks the probes | The probe addresses are not allowed | Allow the addresses in `frazer.ai/probe-ips.json` |
| A heartbeat check fails although the job ran | The job did not call the ping URL, or ran past the period plus the grace | Call the URL at the end of the job; widen the grace or use the job’s cron expression |
| Metrics from a Helm chart are missing | The chart’s `ServiceMonitor` or `PodMonitor` is not installed, or its labels were dropped as high cardinality | Install the chart’s monitors; check the cardinality explorer dashboard |
| No alerts arrive in a channel | The channel is not verified, a silence or maintenance window matches, or the route excludes the severity | Verify the channel and use **Send test**; check silences and windows; check the route |
| A person cannot see a dashboard in another project | Each project is its own tenant, and queries run as the viewer | Grant the person a role in that project, or use **All projects** as an organization admin |

## Reference

### CLI

| Command | What it does |
| --- | --- |
| `frazer logs` | Query a project's logs from the monitoring cell, without access to the cluster. |
| `frazer monitoring metrics` | Query a project's metrics with PromQL from the monitoring cell, without access to the cluster. |
| `frazer monitoring dashboards apply` | Write a dashboard from its view definition (YAML or JSON) to the project's dashboards, with an audit entry; --dry-run shows the diff and the queries' estimated cost. |
| `frazer monitoring dashboards import` | Convert dashboard JSON from another tool into a view definition, with a report per panel. |
| `frazer monitoring dashboards export` | Export a dashboard as Frazer's own YAML or JSON. |

All of them: [CLI reference](https://frazer.ai/docs/cli).

### Events

| Event type | When it is sent |
| --- | --- |
| `frazer.alert.firing.v1` | An alert policy started firing. |
| `frazer.alert.resolved.v1` | A firing alert policy stopped firing. |
| `frazer.incident.opened.v1` | An alert with a new fingerprint opened an incident. |
| `frazer.incident.acknowledged.v1` | A person acknowledged an incident. |
| `frazer.incident.resolved.v1` | An incident was resolved. |
| `frazer.uptime.check.failed.v1` | An uptime check went down under its failure rule. |
| `frazer.uptime.check.recovered.v1` | A failed uptime check is up again. |

All of them: [Events reference](https://frazer.ai/docs/webhooks).

## Related

-   [Frazer Monitoring product page](https://frazer.ai/monitoring)
-   [Security Center](https://frazer.ai/docs/security-center): findings and threats use the same notifier and channels
-   [Identity and access](https://frazer.ai/docs/iam): roles, agents and approvals
-   [Billing and budgets](https://frazer.ai/docs/billing): the monitoring allowance and caps
-   [Webhooks and events](https://frazer.ai/docs/webhooks): signed alert and incident events
-   [Create a Kubernetes cluster](https://frazer.ai/docs/quickstarts/kubernetes-cluster)

## Pages in Operate and secure

- [Frazer Monitoring](https://frazer.ai/docs/monitoring.md) (this page)
- [Security Center](https://frazer.ai/docs/security-center.md)
- [Identity and access](https://frazer.ai/docs/iam.md)
- [Overview](https://frazer.ai/docs/billing.md)