Skip to content

Troubleshooting

This section tracks known issues and their workarounds in the cluster, split into one knowledge-base entry per issue. Each entry follows the same shape: symptom, cause, fix.

Before fixing anything, check known noise and non-remediation: the alerts and symptoms where the obvious corrective action is wrong, the configuration that is deliberately "broken", and the operations that must be escalated rather than attempted. After a change to protected infrastructure, the health verdict is the cross-cutting check.

Symptom ladder

Work down from what you observe to the most likely entry:

  • Secrets / 1Password
  • PushSecret logs spurious HTTP 400 errors but status shows Synced → KB-001
  • GitGuardian flags plaintext credentials or certificate keys in the network-config backup repository → KB-070
  • Flux / GitOps
  • Flood of FluxHelmReleaseArtifactFailed ("flux is OOM"), or ~28 "dependency not ready" alerts while workloads stay healthy → KB-007
  • konflate checks fail on every open PR ("all CIs failing"), or konflate-cache runs out of inodes → KB-011
  • Networking
  • One node's cross-node pod traffic flips/breaks while its host traffic is fine; its spegel pod goes 0/1 → KB-008
  • An app 404s through the gateway on its real hostname but works on its pod IP (live HTTPRoute drifted to *.example.com) → KB-020
  • NodeHighNumberConntrackEntriesUsed on every node at once right after deploying a scanner → KB-023
  • A dozen Gatus endpoints across media/downloads go red at once with HTTP 503, DNS still resolves, pods are simply absent (zeroscaler at 0/1) → KB-027
  • One internal hostname never gets its record (or resolves to the wrong value) while every other new app is fine; the name is already in Unbound as a hand-made row → KB-033
  • Dozens of Flux Kustomizations fail on failed calling webhook ... context deadline exceeded right after a network policy merge, and the webhook answers again seconds later → KB-209
  • OPNsenseDeleteBlocked fires, or opnsense-dns fails every reconcile with row has alias children; refusing to delete → KB-034
  • A certificate on a Traefik instance outside the cluster nears expiry and nothing alerts; ACME renewal has been failing quietly → KB-049
  • Storage / backups
  • Backup pod stuck PodInitializing (mount.nfs: Failed to resolve), or CreateContainerConfigError on a subPath → KB-009
  • After a Rook v1.20 upgrade: RBD nodeplugin FailedCreate, or ~88 VolSyncVolumeOutOfSync alerts → KB-010
  • volsync-system/kopia repository server OOM-crashloops (exit 137) → KB-016
  • CephMonDownQuorumAtRisk (critical) fires minutes after cordoning a control-plane node → KB-019
  • Snapshot CRDs vanish minutes after merging a chart migration; controller crash-loops on "failure to ensure CRDs exist"; HelmRelease rollback loop re-deletes them each retry → KB-029
  • One app's volsync-src-<app>-nfs-* mover pods sit in Error while every other app backs up fine; the log ends write /cache/CACHEDIR.TAG: no space left on device then found existing data in storage location → KB-030
  • A restore hangs on an unbindable cache PVC, is offered less capacity than the data, or hits ENOSPC on /cache, while every backup is green → KB-031
  • The Docker-estate backup reports success every night but the copied data never changes → KB-064
  • The Veeam server is down for days and nothing pages; backup jobs simply stop advancing → KB-067
  • An app's VolSync backups are green but a restore would miss its real data (a sourcePVC patch points at the wrong claim) → KB-145
  • Workloads / pods
  • A JVM/Logstash pod OOMKills on a cadence despite a bounded heap → KB-012
  • A pure-Go pod SIGSEGVs (exit 139) on a large fraction of starts, before any logs → KB-013
  • HelmRelease UpgradeFailed/rollback loop, pod stuck ContainerCreating with a Pulling event (large image) → KB-015
  • allocatable.nvidia.com/gpu = 0 for minutes after a device-plugin swap → KB-014
  • CreateContainerConfigError: runAsUser breaks non-root policy on a fresh render of an s6/LinuxServer image → KB-022
  • One KubeJobFailed a day for netbox-housekeeping (Unknown command: 'housekeeping'), or NetBox housekeeping silently stopped with a scheduled job stuck in the past → KB-032
  • A gluetun VPN sidecar restarts continuously and never reaches Ready, often OOMKilled on every cycle → KB-042
  • SABnzbd's post-processing queue stops advancing while downloads keep completing → KB-059
  • A once-a-day job goes silent for days while /health answers OK and the pod restarts once a day → KB-085
  • An *arr app returns HTTP 401 on the correct password and logs an encryption error → KB-113
  • A web login loops back to the login page in one browser profile while the app logs a successful login, and a private window works → KB-208
  • A custom image crashloops right after a Renovate digest bump that changed its base runtime → KB-157
  • Prowler's worker and beat pods restart in a loop on a Dragonfly authentication error → KB-159
  • Monitoring / Grafana
  • Every panel on one dashboard shows "No data" / "Datasource Prometheus was not found" → KB-021
  • BmcEventLogWarning fires dozens of times for one hardware fault → KB-037
  • Storage-host node and SMART series appear twice, and unscoped queries or mixin rules double-count → KB-038
  • A Gatus endpoint goes down and nothing pages → KB-065
  • Mail relay (smtp2graph)
  • SMTP2GraphQueueStalled holds at one queued message for hours while the canary and later mail deliver fine; the log shows a single Failed to send message and no retries → KB-035
  • Plex playback
  • 4K direct-play freezes for ~60s every ~6 minutes on LAN Apple TVs → KB-002
  • "Server unavailable" / connection drops at session start, pod otherwise healthy → KB-003
  • Remote 4K titles crash with bad lexical cast; the same titles work on phone/LAN → KB-018
  • Apple TV shows one frame of a 4K title then the Plex app freezes (force-quit to recover); the same file plays fine in Infuse → KB-026
  • Talos upgrades
  • TUPPR patch rollout stuck after drain; node cordoned and still on the old version → KB-004
  • Upgrade reports success but the node comes back Ready, uncordoned, and still on the old version; it rebooted promptly → KB-028 (NVRAM wipe left a stale LoaderEntryDefault)
  • Same, but the node took about 5 minutes to go down after the install completed → the reboot-sequence teardown revert in Talos upgrades
  • A node cannot unseal its disk after a BIOS update wipes the Secure Boot keys → KB-207
  • CI / validation / local dev
  • Flate - Test fails or skips on the gpu-operator namespace → KB-005
  • Checkov CKV_K8S_21 flags a namespaced resource as default → KB-006
  • First commit after a mise tool bump dies with ln -sf ... File exists → KB-017

All entries