Observability

Self-hosted metrics, RUM, and tracing for knext — Prometheus, Web Vitals, and OpenTelemetry, with no SaaS lock-in.

knext is deliberately self-hosted: it matches Vercel's compute layer, not its proprietary observability SaaS. Everything here exports to a stack you run — Prometheus, Grafana, and an OTLP backend like Grafana Tempo. No data leaves your cluster, and there is no hosted default.

Observability is configured per app under observability in your knext config. Metrics are always on; RUM and tracing are opt-in and default OFF.

kn-next.config.ts

observability: {
  enabled: true,
  prometheus: { scrapeInterval: '15s' },   // default 15s
  grafana: { enabled: true },              // deploy dashboard ConfigMaps (default true)
  rum: { enabled: false, sampleRate: 1 },  // Web Vitals — default OFF
  tracing: { enabled: false },             // OpenTelemetry — default OFF
}

Metrics (Prometheus)

The runtime exposes a Prometheus endpoint on port 9464, at /metrics — separate from the app's $PORT/3000. That is the port knext ships scrape configuration for: the pod is annotated with prometheus.io/port: "9464" and prometheus.io/path: /metrics, and the shipped PodMonitor targets nothing else. The compiled executable serves it in-process; there is no sidecar.

Why 9464 and not the more common 9091: on a stock Knative Serving install (default config-observability), the queue-proxy sidecar binds :9091 in every revision pod for its own user-metrics server, so an app defaulting to 9091 loses the port race and crash-loops with EADDRINUSE. 9464 is the conventional OpenTelemetry Prometheus-exporter port and collides with nothing Knative runs in the pod. The port is overridable via the METRICS_PORT env — but never set it to 9091, and note the default NetworkPolicy's scrape grants are fixed at 9464 (an app that moves the port opts out of the shipped scrape path). kn-next doctor flags any app pinned onto 9091 while queue-proxy's user metrics are active.

Your app may additionally serve its own GET /api/metrics route — that is where the RUM Web Vitals histograms below land. It is a route in your application, not part of the runtime, and the default scrape does not reach it.

The series the shipped scrape exposes:

Metric	Type	Meaning
`knext_bunexec_http_requests_total`	Counter	Requests served, labelled `status_class` (2xx/3xx/4xx/5xx) — request and error rate.
`knext_bunexec_http_request_duration_seconds`	Histogram	Request latency (buckets down to 5 ms — process boot is ~61 ms).
`knext_bunexec_http_inflight_requests`	Gauge	Current concurrency (saturation).
`knext_bunexec_startup_duration_seconds`	Gauge	How long this pod took to become ready.
`knext_bunexec_process_resident_memory_bytes`	Gauge	Resident set size.
`knext_bunexec_process_uptime_seconds`	Gauge	Seconds since the process started.

Ready-made Grafana dashboards ship with knext (see Dashboards below).

There is no bytecode-cache metric family any more, because there is no cache to be warm or cold about: the compiled executable bakes its bytecode into the binary at build time, so every start reads it — knext_bunexec_startup_duration_seconds measures the result directly.

Dashboards

knext ships three ready-made Grafana dashboards. You do not import them by hand — they are packaged as ConfigMaps labeled grafana_dashboard: "1", the standard label a Grafana sidecar (kube-prometheus-stack, the Grafana Helm chart) watches to auto-load dashboards.

Install them with one command:

kubectl apply -k config/grafana

They land in the monitoring namespace by default. If your Grafana sidecar watches a different namespace, apply into it (kubectl apply -k config/grafana -n <ns>) or edit the overlay's namespace: field. The dashboards then graph the per-app metrics your Prometheus already scrapes from port 9464.

Dashboard	What it shows
RED service health	Request rate, error rate (5xx %), duration percentiles, and in-flight requests — the golden signals.
Scale-to-zero lifecycle	Replica count 0→N, startup durations across pods, and Postgres 0→1 wake rate and latency by pool role.
Load testing	Request throughput and latency under a k6 run.

Two former dashboards were removed rather than left blank: the bytecode cache board (its subject no longer exists — bytecode ships inside the compiled executable) and the RUM — Web Vitals board (its series are collected on the app's own metrics route, which the default scrape does not reach — see the honesty note under RUM). A dashboard whose every panel reads "no data" looks like a quiet system, which is worse than its absence.

Pick your datasource, once. Every dashboard uses a Grafana datasource template variable (no hardcoded datasource UID), so they work against any Prometheus datasource — just select it from the Datasource dropdown at the top of the dashboard the first time you open it.

The replica-count panel on the scale-to-zero dashboard reads kube_deployment_status_replicas, which comes from kube-state-metrics (the cluster), not from knext — so that panel needs kube-state-metrics scraped by the same Prometheus. Every other panel reads a knext_* series exported on 9464.

RUM (Web Vitals)

Real-User Monitoring is opt-in via observability.rum:

rum: { enabled: true, sampleRate: 0.25 }   // enabled is required; sampleRate 0..1, default 1

When enabled, knext sets NEXT_PUBLIC_RUM_ENABLED=true (and, if sampleRate is set, NEXT_PUBLIC_RUM_SAMPLE_RATE) for your app. The client then collects LCP, INP, CLS, FCP, TTFB via Next.js useReportWebVitals and beacons each to the same-origin POST /api/rum endpoint (navigator.sendBeacon, with a fetch+keepalive fallback). With RUM disabled the client sends nothing.

The ingest endpoint records into kn_next_web_vitals_* histograms, merged into /api/metrics. Each carries three labels: app, route, and rating.

These series are not on the default scrape. The shipped Prometheus configuration scrapes the runtime's metrics port (9464), and the Web Vitals histograms live on the app's own /api/metrics route — so out of the box they are collected but not stored. To use RUM data, add a scrape target for your app's /api/metrics route to your Prometheus.

Why a public beacon is safe

A browser beacon cannot carry a Bearer token, so /api/rum is not secured by auth. Instead it is a bounded, fixed-schema aggregator — its only possible effect is observe() on one of a closed set of pre-declared histograms. It is neutered by four independent layers:

Same-origin / cluster-local. Reachable only as broadly as the app itself — governed by the default-on NetworkPolicy. No new external surface.
Fixed-schema lossy aggregator. It cannot create series, set arbitrary values, write storage, or trigger cache revalidation. The worst an attacker can do is skew aggregate percentiles.
Server-enforced bounded cardinality. metric ∈ {LCP, INP, CLS, FCP, TTFB}, rating ∈ {good, needs-improvement, poor}, and route is a server-mapped template — the reported pathname is matched against a closed known-route table; anything unmatched collapses to a single other bucket. Raw paths, UUIDs, and query strings can never become labels. The app label comes from the server environment, never the client.
Rate-limit + size cap + strict shape. An in-process token-bucket limiter caps the rate (429 on flood), a 2 KB payload cap returns 413, and strict allow-list validation returns 400. There is intentionally no GET handler.

Responses: 204 recorded · 400 malformed · 413 oversized · 429 rate-limited.

Tracing (OpenTelemetry)

Distributed tracing is opt-in via observability.tracing and default OFF:

tracing: {
  enabled: true,
  endpoint: 'http://otel-collector.monitoring:4317',  // OTLP/gRPC; default used if unset
  sampleRate: 0.1,                                     // head-based, 0..1, default 1
}

When enabled, knext sets OTEL_TRACING_ENABLED=true (plus OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER_ARG from your config). Your app then exports spans over OTLP to a self-hostable collector. Knative resource attributes (knative.revision, knative.service, knative.configuration, host.name) are attached automatically when present.

No SaaS exporter default. The recommended backend is Grafana Tempo (shares your Grafana, trace→metric exemplars are first-class); Jaeger is the alternative. SaaS exporters (Honeycomb, Datadog, and similar) are not used as a default — they reintroduce lock-in. You may still point endpoint at any OTLP backend you run.

When tracing is disabled, the instrumentation hook returns without initializing OpenTelemetry — no exporter, no span processors, zero overhead.

Load testing

knext ships a k6 load-test harness invoked with kn-next loadtest:

kn-next loadtest --url https://app.example.com --type scale-to-zero --namespace default

It generates a Kubernetes ConfigMap + Job (image grafana/k6) that runs the k6 script in-cluster against your Knative service URL, and cleans itself up via ttlSecondsAfterFinished. Four scenarios are available with --type:

Type	Profile
`smoke`	1 VU for 1m — sanity check.
`load`	ramp to 50 VUs, hold, ramp down.
`spike`	burst to 200 VUs and back.
`scale-to-zero`	a burst, wait past the scale-to-zero window, then a second burst — exercises a cold start.

When observability.enabled is set, k6 results are exported to the in-cluster Prometheus via experimental remote-write.

Load testing is a manual / nightly runbook, not part of your deploy pipeline. It applies an ephemeral Job and never mutates your app's deployment.