Skip to content

Troubleshooting

This section tracks known issues and their workarounds in the cluster, split into one knowledge-base entry per issue. Each entry follows the same shape: Symptom → Cause → Fix.

Symptom ladder

Work down from what you observe to the most likely entry:

  • Secrets / 1Password
    • PushSecret logs spurious HTTP 400 errors but status shows Synced → KB-001
  • Flux / GitOps
    • Flood of FluxHelmReleaseArtifactFailed ("flux is OOM"), or ~28 "dependency not ready" alerts while workloads stay healthy → KB-007
    • konflate checks fail on every open PR ("all CIs failing"), or konflate-cache runs out of inodes → KB-011
  • Networking
    • One node's cross-node pod traffic flips/breaks while its host traffic is fine; its spegel pod goes 0/1 → KB-008
    • An app 404s through the gateway on its real hostname but works on its pod IP (live HTTPRoute drifted to *.example.com) → KB-020
    • NodeHighNumberConntrackEntriesUsed on every node at once right after deploying a scanner → KB-023
    • A dozen Gatus endpoints across media/downloads go red at once with HTTP 503, DNS still resolves, pods are simply absent (zeroscaler at 0/1) → KB-027
  • Storage / backups
    • Backup pod stuck PodInitializing (mount.nfs: Failed to resolve), or CreateContainerConfigError on a subPath → KB-009
    • After a Rook v1.20 upgrade: RBD nodeplugin FailedCreate, or ~88 VolSyncVolumeOutOfSync alerts → KB-010
    • volsync-system/kopia repo server OOM-crashloops (exit 137) → KB-016
    • CephMonDownQuorumAtRisk (critical) fires minutes after cordoning a control-plane node → KB-019
  • Workloads / pods
    • A JVM/Logstash pod OOMKills on a cadence despite a bounded heap → KB-012
    • A pure-Go pod SIGSEGVs (exit 139) on a large fraction of starts, before any logs → KB-013
    • HelmRelease UpgradeFailed/rollback loop, pod stuck ContainerCreating with a Pulling event (large image) → KB-015
    • allocatable.nvidia.com/gpu = 0 for minutes after a device-plugin swap → KB-014
    • CreateContainerConfigError: runAsUser breaks non-root policy on a fresh render of an s6/LinuxServer image → KB-022
  • Monitoring / Grafana
    • Every panel on one dashboard shows "No data" / "Datasource Prometheus was not found" → KB-021
  • Plex playback
    • 4K direct-play freezes for ~60s every ~6 minutes on LAN Apple TVs → KB-002
    • "Server unavailable" / connection drops at session start, pod otherwise healthy → KB-003
    • Remote 4K titles crash with bad lexical cast; the same titles work on phone/LAN → KB-018
    • Apple TV shows one frame of a 4K title then the Plex app freezes (force-quit to recover); the same file plays fine in Infuse → KB-026
  • Talos upgrades
    • TUPPR patch rollout stuck after drain; node cordoned and still on the old version → KB-004
    • Upgrade reports success but the node comes back Ready, uncordoned, and still on the old version (after a BIOS flash) → KB-028
  • CI / validation / local dev
    • Flate - Test fails or skips on the gpu-operator namespace → KB-005
    • Checkov CKV_K8S_21 flags a namespaced resource as default → KB-006
    • First commit after a mise tool bump dies with ln -sf ... File exists → KB-017

All entries