Skip to content

opnsensegh

Purpose

kubernetes/apps/observability/opnsensegh runs a second copy of the API-based OPNsense exporter against another OPNsense firewall, under its own OPNSENSE_EXPORTER_INSTANCE_LABEL, so the existing OPNsense dashboard lists it in its instance selector without a dashboard of its own.

How the scrape is routed

The target address is GH_OPNSENSE_ADDR in cluster-secrets. Pods reach it through the main firewall's site-to-site IPsec tunnel (network-ops docs/runbooks/gardenhouse-ipsec.md): the cluster's traffic leaves via its default gateway and the tunnel's policy captures it, so no proxy or Tailscale route is involved. This replaced a Tailscale operator egress proxy, which could only reach the firewall through a DERP relay because it sits behind carrier-grade NAT.

If both exporters stop scraping at once, check the tunnel first: OPNsenseIPsecTunnelDown on the main firewall covers it, and the scrape itself rides it.

OS-level metrics

opnsensegh-node is a ScrapeConfig for node_exporter on the firewall (port 9100), the same shape as the main firewall's opnsense-node described in opnsense-exporter. It is scraped directly at GH_OPNSENSE_ADDR over the site-to-site tunnel, and relabelled to instance="opnsense-gh" so it joins the API exporter's series. It drops node_uname_info for the same reason as the other node scrapes. The target stays down until node_exporter is installed on the firewall.

Config backup

The firewall's encrypted config is backed up daily by the second CronJob of opnsense-config-backup, using the opnsensegh-config-backup 1Password item.

Alerts

OPNsenseGhExporterDown fires after 10 minutes of a failed scrape, a vanished target, or a missing WAN gateway series (the exporter answers but its API call to the firewall fails). It does not use opnsense_up: exporter v0.0.17 reads this firewall's string "OK" system status as down, so that series sits at 0 while every endpoint works (upstream #119). Revisit once a release fixes it. Check the tunnel (OPNsenseIPsecTunnelDown) before reading it as the firewall being down. OPNsenseGhWanLossHigh reuses the main firewall's 20% loss threshold. Both are warning: the value of this app is the history.

Speedtest metrics

kubernetes/apps/observability/opnsenseghspeedtest exposes the hourly Ookla result that the firewall's speedtest plugin records. It is a separate app because the exporter above has no speedtest collector and a second controller in this HelmRelease would share its ServiceMonitor selector.

A json_exporter pod is probed every 5 minutes by a Prometheus Probe, and fetches the plugin's showrecent endpoint over the same IPsec tunnel. The module config is rendered by the app's ExternalSecret from the opnsensegh-exporter item, because json_exporter can only take basic-auth credentials from its config file.

  • Metrics. opnsense_speedtest_download_mbps, opnsense_speedtest_upload_mbps and opnsense_speedtest_latency_ms are gauges. The API returns them as JSON strings, which json_exporter parses as floats.
  • Test time. json_exporter only reads a numeric epoch for timestamps, and this API returns an ISO date string, so the time is the date label (UTC, no zone) on opnsense_speedtest_last_info.
  • No alerts. There are no thresholds yet; the Speedtest dashboard shows throughput and latency history.

OPNsenseGhSpeedtestStale fires when opnsense_speedtest_samples (the stored result count from showstat) has not changed for 3 hours, two missed hourly runs in a row. The showrecent probe keeps re-serving the last good result if the firewall's cron job dies, so probe success alone proves nothing.