Skip to content

NetBox

Purpose

NetBox provides DCIM (data centre infrastructure management) and IPAM (IP address management): the source of truth for devices, racks, prefixes, and IP allocations. It runs internal-only in the default namespace: a web frontend and an RQ background worker (webhooks, scripts, reports, and the daily housekeeping system job).

Design decisions

NetBox is one of the few apps here that does not use the bjw-s app-template chart. It uses the official netbox-chart (oci://ghcr.io/netbox-community/netbox-chart/netbox) via an OCIRepository + chartRef, because NetBox's multi-workload topology (web + worker) and its existingSecret contract are already modelled by the upstream chart.

  • Namespace default, modelled on the in-repo paperless app (its closest twin).
  • PostgreSQL: shared CNPG cluster (postgres18-rw.database.svc.cluster.local, sslmode=require); bundled Postgres disabled. The netbox role and database are declared as CNPG DatabaseRole and Database resources in kubernetes/apps/database/cloudnative-pg/tenants/netbox.yaml; no init container and no superuser credential in the app Secret.
  • Redis: external Dragonfly (dragonfly.database.svc.cluster.local:6379); the chart's bundled cache (valkey.enabled) is disabled. Separate logical DB indices for the tasks queue (0) vs the cache (1).
  • Media: a local ceph-block PVC consumed via persistence.existingClaim and backed up by the VolSync component (VOLSYNC_CAPACITY). NetBox uses no object storage; the cluster's S3 tier is Garage (storage namespace), not Ceph RGW (which was removed).
  • Auth / exposure: superuser-only first pass (pocket-id OIDC is a fast-follow); internal-only HTTPRoute on envoy-internal, never publicly exposed.
  • Health monitored automatically by the gatus-sidecar (it auto-discovers the HTTPRoute); reloader.stakater.com/auto rolls pods on secret/config change; a ServiceMonitor exposes metrics to kube-prometheus-stack.

Deploy gotchas

  • One shared netbox-secret must carry the chart's EXACT key names. All of the chart's secret references (global existingSecret, superuser.existingSecret, externalDatabase / tasksDatabase / cachingDatabase existingSecretName) point at the single netbox-secret, so that Secret must contain every key the chart projects, spelled exactly as the chart expects:
  • secret_key, email_password (the chart's projected secrets volume requires the key to exist even when email is unused, so set it to an empty string).
  • password and api_token (superuser).
  • db_password (the value of externalDatabase.existingSecretKey).
  • tasks_password and cache_password (the Dragonfly tasks/caching DB keys).

  • These mismatches do not show up in a flate (local) or Konflate (in-cluster PR render): the manifest is valid. They only fail at runtime as pod FailedMount (references non-existent secret key) or CreateContainerConfigError (couldn't find key …). Before wiring the ExternalSecret, render the chart and grep the output for every secretKeyRef key: and projected-volume items[].key:

    flate build hr netbox -n default --path kubernetes/flux/cluster

  • Web and worker share one RWO media PVC, so they must colocate. With a ReadWriteOnce ceph-block claim, scheduling the pods onto different nodes deadlocks on Multi-Attach during a rollout. Pin them together with podAffinity (anchor on the worker) or hit a stuck rollout.

  • The chart's housekeeping CronJob is disabled, deliberately. v4.7.0 removed manage.py housekeeping, the command that CronJob runs, and moved the work into an internal RQ system job that manage.py rqworker schedules itself, so the worker is now the only component housekeeping needs. The chart has not caught up and still defaults it on, which costs a daily KubeJobFailed. Do not "fix" this by deploying an rqscheduler — see KB-032, which also covers how the internal job wedges permanently and how to un-wedge it.
  • Granian web OOMs at the chart's default 4 workers (peaks ~956Mi against a 1Gi limit). Set GRANIAN_WORKERS=2 and give the web pod request 512Mi / limit 1.5Gi.
  • Django host guarding rejects unlisted Host headers, including kubelet probes. ALLOWED_HOSTS must include both the route host(s) and the in-cluster service name (the chart's allowedHostsIncludesPodIP covers the pod IP), and CSRF trusted origins must list https://netbox.${SECRET_DOMAIN}. Miss this and probes (and the UI) get a 400.
  • SECRET_KEY must be 50+ characters. Generate it long enough (e.g. openssl rand -base64 60) or NetBox refuses to start.
  • The 1Password source item lives in the Talos vault and is created via the op CLI (never the UI), with values generated locally.

Operational notes

  • Reconcile lives in the default namespace; it dependsOn cloudnative-pg-tenants and dragonfly-cluster in the database namespace, so reconcile those first if NetBox shows "dependency not ready".
flux reconcile kustomization netbox -n default
  • Hostname is netbox.${SECRET_DOMAIN} (internal DNS via external-dns). The HTTPRoute's only hostname sits on the envoy-internal listener, which is what keeps it internal-only. It does not expose NetBox publicly.
  • First-boot check: confirm kubectl -n database get databaserole,database netbox shows APPLIED true for both, then log in as the superuser (admin).
  • Backups: the media PVC is snapshotted by VolSync. The netbox database lives on the shared CNPG cluster and is backed up by CNPG's own path, not VolSync.
  • Teardown: prune: true removes NetBox's Kubernetes resources on revert, but the netbox database on the shared CNPG cluster and the 1Password item persist and must be dropped manually for a clean teardown.