Observability

Self-hosted metrics, RUM, and tracing for knext — Prometheus, Web Vitals, and OpenTelemetry, with no SaaS lock-in.

knext is deliberately self-hosted: it matches Vercel's compute layer, not its proprietary observability SaaS. Everything here exports to a stack you run — Prometheus, Grafana, and an OTLP backend like Grafana Tempo. No data leaves your cluster, and there is no hosted default.

Observability is configured per app under observability in your knext config. Metrics are always on; RUM and tracing are opt-in and default OFF.

knext.config.ts

observability: {
  enabled: true,
  prometheus: { scrapeInterval: '15s' },   // default 15s
  grafana: { enabled: true },              // deploy dashboard ConfigMaps (default true)
  rum: { enabled: false, sampleRate: 1 },  // Web Vitals — default OFF
  tracing: { enabled: false },             // OpenTelemetry — default OFF
}

Metrics (Prometheus)

The runtime exposes a Prometheus endpoint on port 9464, at /metrics — separate from the app's $PORT/3000. That is the port knext ships scrape configuration for: the pod is annotated with prometheus.io/port: "9464" and prometheus.io/path: /metrics, and the shipped PodMonitor targets nothing else. On the vinext build the server entry serves it in-process. On the default standalone build the supervisor that starts Next's server.js serves it.

Why 9464 and not the more common 9091: on a stock Knative Serving install (default config-observability), the queue-proxy sidecar binds :9091 in every revision pod for its own user-metrics server, so an app defaulting to 9091 loses the port race and crash-loops with EADDRINUSE. 9464 is the conventional OpenTelemetry Prometheus-exporter port and collides with nothing Knative runs in the pod. The port is overridable via the METRICS_PORT env — but never set it to 9091, and note the default NetworkPolicy's scrape grants are fixed at 9464 (an app that moves the port opts out of the shipped scrape path). knext doctor flags any app pinned onto 9091 while queue-proxy's user metrics are active.

Your app may additionally serve its own GET /api/metrics route — that is where the RUM Web Vitals histograms below land. It is a route in your application, not part of the runtime, and the default scrape does not reach it.

The series depend on the build. The vinext build exposes:

Metric	Type	Meaning
`knext_bunexec_http_requests_total`	Counter	Requests served, labelled `status_class` (2xx/3xx/4xx/5xx) — request and error rate.
`knext_bunexec_http_request_duration_seconds`	Histogram	Request latency (buckets down to 5 ms — a local warm-binary process boot measured ~61 ms; on a real cluster, pod scheduling and container start dominate the actual wake time — see Scale-to-zero & cold starts).
`knext_bunexec_http_inflight_requests`	Gauge	Current concurrency (saturation).
`knext_bunexec_startup_duration_seconds`	Gauge	How long this pod took to become ready.
`knext_bunexec_process_resident_memory_bytes`	Gauge	Resident set size.
`knext_bunexec_process_uptime_seconds`	Gauge	Seconds since the process started.

The default standalone build exposes a different family, with an app label on every series:

Metric	Type	Meaning
`knext_http_requests_total`	Counter	Requests served, labelled `method` and `status_class`.
`knext_http_request_duration_seconds`	Histogram	Request latency.
`knext_http_inflight_requests`	Gauge	Current concurrency (saturation).
`knext_coldstart_total`, `knext_coldstart_duration_seconds`	Counter, Histogram	Cold-start count and wake latency.
`knext_db_wake_total`, `knext_db_wake_duration_seconds`	Counter, Histogram	Database wake count and latency, labelled `role`.
`knext_deep_health_state`	Gauge	The deep-health verdict per dependency; the active state is `1`. Served with tracing on or off, and present only while the app has used its database in the last 45 seconds (an idle app serves no such series); the only series of this family the shipped scrape collects, and the one the deep-health alerts read.

Ready-made Grafana dashboards ship with knext (see Dashboards below). The request panels on the RED and load-testing dashboards query the vinext family (knext_bunexec_*), so on the default standalone build point them at the knext_http_* series instead.

There is no bytecode-cache metric family any more, because there is no cache to be warm or cold about: the compiled executable bakes its bytecode into the binary at build time, so every start reads it — knext_bunexec_startup_duration_seconds measures the result directly.

Existing apps: start the deep-health gauge

A new app starts the deep-health gauge with tracing off by itself. An app you created earlier gets it only after both of these: update @getknext/core to the release that ships it, and add the lines below to its own src/instrumentation.ts (or regenerate that file from the scaffold). Until then, with tracing off, the app serves no deep-health series and the two deep-health alerts stay silent.

In register(), change the tracing-off branch (the one that returns before importing ./instrumentation-node) to start the gauge before it returns:

  if (!resolveOtelOptions(process.env)) {
    void import('@getknext/core/internal/deep-health-lite')
      .then(({ startDeepHealthLite }) => startDeepHealthLite())
      .catch(() => {});
    return;
  }

Keep it after the NEXT_RUNTIME !== 'nodejs' check, and do not await it, so it adds nothing to the start-up path. An app with tracing on needs no change. The shipped scrape forwards the series from your app's runtime-contract.mjs; if that file predates the deep-health forward, regenerate it as well.

Dashboards

knext ships three ready-made Grafana dashboards. You do not import them by hand — they are packaged as ConfigMaps labeled grafana_dashboard: "1", the standard label a Grafana sidecar (kube-prometheus-stack, the Grafana Helm chart) watches to auto-load dashboards.

Install them with one command:

kubectl apply -k config/grafana

They land in the monitoring namespace by default. If your Grafana sidecar watches a different namespace, apply into it (kubectl apply -k config/grafana -n <ns>) or edit the overlay's namespace: field. The dashboards then graph the per-app metrics your Prometheus already scrapes from port 9464.

Dashboard	What it shows
RED service health	Request rate, error rate (5xx %), duration percentiles, and in-flight requests — the golden signals.
Scale-to-zero lifecycle	Replica count 0→N, startup durations across pods, and Postgres 0→1 wake rate and latency by pool role.
Load testing	Request throughput and latency under a k6 run.

Two former dashboards were removed rather than left blank: the bytecode cache board (its subject no longer exists — bytecode ships inside the compiled executable) and the RUM — Web Vitals board (its series are collected on the app's own metrics route, which the default scrape does not reach — see the honesty note under RUM). A dashboard whose every panel reads "no data" looks like a quiet system, which is worse than its absence.

Pick your datasource, once. Every dashboard uses a Grafana datasource template variable (no hardcoded datasource UID), so they work against any Prometheus datasource — just select it from the Datasource dropdown at the top of the dashboard the first time you open it.

The replica-count panel on the scale-to-zero dashboard reads kube_deployment_status_replicas, which comes from kube-state-metrics (the cluster), not from knext — so that panel needs kube-state-metrics scraped by the same Prometheus. Every other panel reads a knext_* series exported on 9464.

RUM (Web Vitals)

Real-User Monitoring is opt-in via observability.rum:

rum: { enabled: true, sampleRate: 0.25 }   // enabled is required; sampleRate 0..1, default 1

When enabled, knext sets NEXT_PUBLIC_RUM_ENABLED=true (and, if sampleRate is set, NEXT_PUBLIC_RUM_SAMPLE_RATE) for your app. The client then collects LCP, INP, CLS, FCP, TTFB via Next.js useReportWebVitals and beacons each to the same-origin POST /api/rum endpoint (navigator.sendBeacon, with a fetch+keepalive fallback). With RUM disabled the client sends nothing.

The ingest endpoint records into kn_next_web_vitals_* histograms, merged into /api/metrics. Each carries three labels: app, route, and rating.

These series are not on the default scrape. The shipped Prometheus configuration scrapes the runtime's metrics port (9464), and the Web Vitals histograms live on the app's own /api/metrics route — so out of the box they are collected but not stored. To use RUM data, add a scrape target for your app's /api/metrics route to your Prometheus.

Why a public beacon is safe

A browser beacon cannot carry a Bearer token, so /api/rum is not secured by auth. Instead it is a bounded, fixed-schema aggregator — its only possible effect is observe() on one of a closed set of pre-declared histograms. It is neutered by four independent layers:

Same-origin / cluster-local. Reachable only as broadly as the app itself — governed by the default-on NetworkPolicy. No new external surface.
Fixed-schema lossy aggregator. It cannot create series, set arbitrary values, write storage, or trigger cache revalidation. The worst an attacker can do is skew aggregate percentiles.
Server-enforced bounded cardinality. metric ∈ {LCP, INP, CLS, FCP, TTFB}, rating ∈ {good, needs-improvement, poor}, and route is a server-mapped template — the reported pathname is matched against a closed known-route table; anything unmatched collapses to a single other bucket. Raw paths, UUIDs, and query strings can never become labels. The app label comes from the server environment, never the client.
Rate-limit + size cap + strict shape. An in-process token-bucket limiter caps the rate (429 on flood), a 2 KB payload cap returns 413, and strict allow-list validation returns 400. There is intentionally no GET handler.

Responses: 204 recorded · 400 malformed · 413 oversized · 429 rate-limited.

Tracing (OpenTelemetry)

Distributed tracing is opt-in via observability.tracing and default OFF:

tracing: {
  enabled: true,
  endpoint: 'http://otel-collector.monitoring:4317',  // OTLP/gRPC; default used if unset
  sampleRate: 0.1,                                     // head-based, 0..1, default 1
}

When enabled, knext sets OTEL_TRACING_ENABLED=true (plus OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER_ARG from your config). Your app then exports spans over OTLP to a self-hostable collector. Knative resource attributes (knative.revision, knative.service, knative.configuration, host.name) are attached automatically when present.

No SaaS exporter default. The recommended backend is Grafana Tempo (shares your Grafana, trace→metric exemplars are first-class); Jaeger is the alternative. SaaS exporters (Honeycomb, Datadog, and similar) are not used as a default — they reintroduce lock-in. You may still point endpoint at any OTLP backend you run.

When tracing is disabled, the instrumentation hook returns before it loads any OpenTelemetry code — no exporter, no span processors, and the tracing modules are never imported, so a cold start does not pay to load them. Turning tracing on is still only the config above: the app loads them at startup when OTEL_TRACING_ENABLED=true is present.

Apps created with an earlier knext create check the tracing switch only after importing src/instrumentation-node.ts, which loads the whole tracing stack on every cold start even with tracing off. To get the faster start, add the same check to src/instrumentation.ts, before that import:

import { resolveOtelOptions } from '@getknext/core/adapters/otel-config';

export async function register() {
  if (process.env.NEXT_RUNTIME !== 'nodejs') return;
  if (!resolveOtelOptions(process.env)) return; // tracing off: load nothing
  const { registerNode } = await import('./instrumentation-node');
  registerNode();
}

Load testing

knext ships a k6 load-test harness. It is a directly runnable entry, not a knext subcommand (see Directly runnable entries):

node node_modules/@getknext/core/dist/cli/loadtest.js --url https://app.example.com --type scale-to-zero --namespace default

It generates a Kubernetes ConfigMap + Job (image grafana/k6) that runs the k6 script in-cluster against your Knative service URL, and cleans itself up via ttlSecondsAfterFinished. Four scenarios are available with --type:

Type	Profile
`smoke`	1 VU for 1m — sanity check.
`load`	ramp to 50 VUs, hold, ramp down.
`spike`	burst to 200 VUs and back.
`scale-to-zero`	a burst, wait past the scale-to-zero window, then a second burst — exercises a cold start.

When observability.enabled is set, k6 results are exported to the in-cluster Prometheus via experimental remote-write.

Load testing is a manual / nightly runbook, not part of your deploy pipeline. It applies an ephemeral Job and never mutates your app's deployment.