joryirving/home-ops adoption roadmap¶

Status: Largely SHIPPED 2026-07-10. PRs A-E landed as #3480-#3484 (+ follow-up fixes #3488 mcp-searxng bind / arr telemetry, #3492 cache-prep securityContext), seerr MCP as #3491, and PR G as the staged #3489 (operator 0.9.3) → #3490 (shared cache + abliterated model) → #3493 (vision, first-class mmproj) with both models verified on the shared CephFS cache (text + vision end-to-end). The CephFS RWX smoke test passed on 2026-07-10. Still open: ha MCP (awaiting a user-created read-only Home Assistant token in 1Password), PR F (optional CPU auxiliary model), and foreman (trialled 2026-08-25, then archived, see below). Planned 2026-07-09 via multi-agent gap analysis + adversarial review.

A survey of joryirving/home-ops (kubernetes/apps/base/llm and .agents) against this cluster. Most of his stack was already ported (open-webui, litellm, llmkube, memini, toolhive); this roadmap covers the worthwhile delta, two load-bearing findings that came out of verification, and the plan to unwind workarounds built on a wrong premise.

Load-bearing findings¶

1. CephFS was never blocked on this cluster. The June 2026 "Talos ships no ceph kernel module" diagnosis was a modprobe-vs-builtin trap: CONFIG_CEPH_FS=y is compiled into the Talos v1.13.5 kernel and /proc/filesystems registers ceph on all three nodes today. See KB-025. This invalidates the premise behind the RWO model-storage workarounds (PR G below) and un-blocks foreman (see below).
2. Cross-namespace CiliumNetworkPolicy is the hidden prerequisite for new MCP servers. The media, downloads, and home namespaces only allow ingress from network (and, for media, from downloads). Any MCP server pod in ai calling those services directly needs a new ingress rule on each target namespace (precedent: allow-litellm-from-consumers in kubernetes/apps/ai/netpol.yaml).
3. Hardware verdict for the "ryzen" question: the nodes' AMD Ryzen 7000-class CPUs carry a 2-CU display-class iGPU sharing the CPUs' DDR5: nothing like the Strix-Halo-class AMD hardware jory serves models on (and his llama-ryzen runs on a separate utility cluster anyway). The iGPU path is rejected. The useful Ryzen angle is CPU serving of small auxiliary models (Zen 4 AVX-512, large RAM, nodes at 6-8% utilisation), which the cluster already proves with llama-embed / llama-rerank on the plain CPU llama.cpp:server image.

The PR ladder¶

Ordered so each PR is independently mergeable; nothing after PR A depends on anything before it except where noted.

PR A: LLM observability¶

• kubernetes/apps/ai/litellm/app/prometheusrule.yaml: port jory's four alerts (LiteLLMFallbackChainExhausted critical, LiteLLMModelFailover, LiteLLMDeploymentOutage, LiteLLMAuthOrQuotaFailures) onto the litellm_deployment_* series the existing ServiceMonitor already scrapes. Tune thresholds for a single replica, and use absent() / for: windows: several series only appear after a first failure. Consider folding in a direct InferenceService-down alert (pod availability on metrics already scraped); the review pass flagged it as a comparable cheap win.
• ToolHive telemetry: an MCPTelemetryConfig CR (prometheus.enabled: true) in ai, plus a PodMonitor for the MCP server pods. Drop jory's release: kube-prometheus-stack selector label: this cluster's Prometheus selects monitors without it, and copying it could silently break selection.
• llama.cpp serving dashboard (jory's llama-server.json) as a GrafanaDashboard, plus targetLabels: [app.kubernetes.io/instance, app.kubernetes.io/name] on the llmkube-models ServiceMonitor so panels can filter per backend. Datasource name must be lowercase prometheus (KB-021). Optionally set prometheus.inferencePodMonitor.enabled: true in the llmkube HelmRelease (jory has it on; ours is off).

PR B: config cherry-picks¶

• SearXNG (toolhive/mcp-servers/searxng): disable autocomplete and favicon_resolver (per-keystroke upstream calls trip captcha / rate limits on an automation-driven instance; ours is driven by the MCP server and open-webui RAG), add jory's enabled_plugins set and hostname priority boosting, add a limiter.toml with RFC1918 pass_ip. Skip the Redis-backed limiter, replicas: 2, and the Brave API engine.
• litellm router_settings.context_window_fallbacks: oversized prompts to the 32k self-hosted model fall back to the larger-context model instead of erroring.
• .agents/skills/add-app/SKILL.md: reword the hardcoded AskUserQuestion reference to host-neutral phrasing (name it as the Claude Code example). Single line.

PR C: MCP servers over existing apps¶

• arr: a ToolHive MCPServer wired to sonarr/radarr (media) and prowlarr (downloads), API keys extracted from the existing 1Password items. Requires new ingress CiliumNetworkPolicy rules on both media and downloads (finding 2). Gate: source a digest-pinnable image: jory runs it via npx at runtime, which violates the image-pinning convention here. Register in the mcp-tools group and litellm mcp_servers.
• ha: MCPServer for Home Assistant with a read-only scoped long-lived token (new 1Password item) and a new ingress rule on home. Treat read-only scoping as mandatory: LLM-driven home control is the highest blast radius in this batch.
• seerr (optional): the review pass called this the weakest "skip": same exploit-an-existing-app class as arr, nearly free once the media netpol rule exists. Add read-only as a follow-on if wanted.
• Alternative to all netpol edits: point the MCP servers at the apps' envoy-internal gateway URLs instead of .svc DNS (traffic then originates from network, which every namespace already allows). Weigh one hop + TLS against three netpol rules.
• Watch the tool budget: litellm's mcp_semantic_tool_filter (top_k 8, threshold 0.3) gets more low-similarity tools to rank; may need retuning.

PR D: CephFS enablement¶

Values-only; never touches the in-use ceph-blockpool / ceph-block StorageClass. Two files:

1. kubernetes/apps/rook-ceph/rook-ceph/csi-drivers/helmrelease.yaml: drivers.cephfs.enabled: true, mirror the RBD block's snapshotter settings, and set kernelMountOptions: ms_mode=prefer-crc: this cluster sets requireMsgr2: true, and the kernel client needs ms_mode to negotiate msgr2. Rewrite the stale "no ceph kernel module" comment (KB-025).
2. kubernetes/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml: replace cephFileSystems: [] with a ceph-filesystem entry: distinct metadata + data0 pools (replicated size 3, failureDomain: host), MDS activeCount: 1 + activeStandby: true (requests ~100m/1Gi, limit 4Gi; decide the ceiling), and the ceph-filesystem StorageClass. Rewrite the comment block. Optionally uncomment the cephFileSystemVolumeSnapshotClass (csi-ceph-filesystem).

No Talos, schematic, or operator changes. Validate with just kube flate-build-hr rook-ceph rook-ceph-cluster, then after merge: MDS pods Ready, ceph -s HEALTH_OK, and the gate for PR G: a throwaway RWX PVC mounted read-write by two pods on different nodes. The June attempt failed at a layer never pinned down, so the smoke test is the insurance before anything real depends on CephFS.

PR E: hermes¶

Stand up NousResearch/hermes-agent at kubernetes/apps/ai/hermes/ as a standard app-template app. All dependencies (litellm, memini, toolhive) are same-namespace: no new netpol. It idles fine with zero bot-token chat platforms, so the initial baseline was gateway + dashboard only. That has since been superseded: hermeswebui (kubernetes/apps/ai/hermeswebui/), a chat web frontend, is now deployed, and hermes sets API_SERVER_ENABLED: "true" specifically to serve it.

Shape (validated against the open-webui HelmRelease as the structural analog):

• ks.yaml: components volsync, dependsOn litellm + memini + onepassword-connect (review catch: the ExternalSecret convention), VOLSYNC_CAPACITY: 10Gi.
• app/configmap.yaml (hermes-config): default model self-hosted via a litellm provider (http://litellm.ai.svc.cluster.local:4000/v1), memory provider memini (MEMINI_URL pointing at the memini Service, distinct MEMINI_NAMESPACE), terminal backend local with /opt/data/workspace, security.redact_secrets + privacy.redact_pii on.
• app/externalsecret.yaml: extract litellm, memini, and a new hermes-agent 1Password item (dashboard basic-auth credentials + session secret; bot tokens later).
• app/helmrelease.yaml: fsGroup: 10000 (image-baked UID, not the usual 1000), init container bootstraps the pinned memini plugin, gateway + dashboard containers, route on envoy-internal for {{ .Release.Name }}.${SECRET_DOMAIN}, persistence existingClaim on /opt/data. Strip everything jory-specific: Authentik OIDC, the NAS mount, Discord persona, code-server sidecar.

Verify before applying (review-pass gaps): the image tag + sha256 digest and container ports (8642 gateway / 9119 dashboard were research-derived, not confirmed in-repository); memini's actual Service port; the real ToolHive MCP proxy Service names (the live litellm ConfigMap disagrees with research snapshots on ports).

Proposed defaults, changeable later; see the decision checklist below for the open ones: no chat platforms at first boot; MCP wired directly per ToolHive proxy; approvals.mode: manual (hermes gets kubectl/flux/talos tools and a local terminal, the Pod is the only boundary, so no unattended cluster actions until it has earned trust); a scoped LiteLLM virtual key from the start, matching the operator-issued LiteLLMVirtualKey pattern open-webui and memini already use, rather than reusing the master key; MEMINI_NAMESPACE: hermes.

PR F: (optional) CPU-served auxiliary model¶

A ~4B Q4 model (for example Qwen3-4B-Instruct Q4_K_M) served CPU-only to handle summaries / classification / drafts without spending an L4 slice. Mirror the proven app-template pattern of llama-embed / llama-rerank (plain ghcr.io/ggml-org/llama.cpp server tag: the CPU build; there is no server-cpu tag) rather than the unverified llmkube hardware.accelerator: cpu path. Expect ~10-18 tok/s generation (dual-channel DDR5 bound). Fine for latency-tolerant work, not for interactive chat. Register in litellm as a distinct model name and consider pointing litellm's auxiliary tasks at it. Set explicit requests/limits to protect co-tenants.

PR G: model-storage de-workaround (requires PR D)¶

The RWO pattern in kubernetes/apps/ai/llmkube/models/ exists only because of the wrong June diagnosis (KB-025): per-model ceph-block PVCs, one-shot curl staging Jobs carrying kustomize.toolkit.fluxcd.io/reconcile: disabled (Job immutability vs Renovate), and modelCache.enabled: false in the llmkube HelmRelease with the wrong comment. Target state is jory's, verbatim from his llmkube HelmRelease:

modelCache:
  enabled: true
  mode: shared
  accessMode: ReadWriteMany
  storageClass: ceph-filesystem
  size: 200Gi # ours: 100Gi is ample for the current ~27Gi of GGUFs + headroom

Steps:

1. Flip modelCache in kubernetes/apps/ai/llmkube/app/helmrelease.yaml to the block above and rewrite its comment.
2. Convert the two Model CRs (qwen3.6-35b-a3b, qwen3-30b-abliterated) from source: pvc://… to operator-managed upstream sources so llmkube stages weights into the shared cache itself (mirror the Model source syntax in jory's manifests, his memini-summary Model pulls unsloth/Qwen3.5-4B-GGUF directly; verify the exact scheme against the chart's CRD docs).
3. Delete the staging Jobs, their reconcile: disabled + Checkov-skip annotations, and (after cutover is verified) the per-model PVCs. Do one model at a time: suspend risk is real (KB-015-style HelmRelease timeout thrash if a Recreate rollout wedges), and the old PVC is the instant rollback until the cache-served pod is healthy.
4. Keep --no-mmap (jory keeps it on CephFS too: cold-fault avoidance). The --mmproj /model-source/mmproj-F16.gguf path will change with the cache mount: verify vision still works end-to-end (the loupe app depends on self-hosted carrying the mmproj projector).
5. No VolSync on the cache: weights are re-downloadable by design.

What it buys: no staging Jobs or immutability hacks, operator-managed model lifecycle, one shared weights copy, and any node can serve any model without re-staging: model switching and failover stop being a re-download event. The llm-gpu-model anti-affinity spread stays.

Foreman: trialled (#4585), archived 2026-08-26¶

Outcome: archived. The manifests moved to .archive/kubernetes/apps/ai/foreman/ and the ./foreman/ks.yaml line left the ai namespace kustomization, so Flux pruned the operator, the three Agent CRs, the foreman-gate-cache PVC (a rebuildable gate cache, nothing to snapshot), the litellm-key-foreman virtual key and the GitHub token generator. What the one hand-applied Workload trial showed:

• The local coder (self-hosted, Qwen3.8-27B on a single L4) managed about 30 model turns an hour, roughly 178k prompt tokens in the hour without producing a diff. A real issue needs 100+ turns, so one attempt is a multi-hour affair on this hardware.
• The first attempt died with a one-off agent Deployment rollout (its claim expired with the pod). That is a fragility of Job-mode execution the retry budget does not cover.
• Foreman reads GITHUB_TOKEN once at pod start, so the hourly installation tokens the GithubAccessToken generator mints expire mid-attempt; a long-lived PAT (jory runs a bot user with one) is the only token shape that survives a full attempt.
• There is no queue of well-specified mechanical issues here for it to work through. Work in this repository arrives as "look at this and figure out what is wrong", which an unattended issue → pull request loop cannot do.

To reinstate: git mv the directory back, re-add the ks.yaml line, and before the next trial (a) point the coder Agent at a cloud model (openrouter/auto) and keep reviewer local, (b) add a self-hosted alias with reasoning_effort: none for any local coding lane, and © recreate the foreman-github 1Password item with a fine-grained PAT and flip the foreman-agent ExternalSecret from the generator back to extract. The foreman-lukeevanstech GitHub App was deleted on 2026-08-26 and the foreman-github item with it (a private key for a deleted App is a liability, not a spare), so the generator path needs a fresh App registration (github.com/settings/apps/new: Contents RW / Pull requests RW / Issues R / Metadata R, webhook off, installed on this repository only) before it works again. The rest of this section is the trial-time write-up, kept for the pick-up.

foreman + dispatch + foreman-dispatch-bridge is jory's autonomous "GitHub issues in → pull requests out" pipeline: dispatch grooms and lanes issues with a small local model, the bridge CronJob claims one ready issue per lane and creates a foreman Workload, and foreman's agent pods (per-language coders → deterministic lint/test gate → read-only reviewer, all inferencing through litellm) open the PR, with a big-context cloud model as the escalation lane.

Its hard blocker here was the gateCache RWX volume, gone once PR D landed. The dispatch + foreman-dispatch-bridge half is still deliberately skipped (this trial hand-applies one Workload at a time); jory's GATEPROFILE_MAP is the bridge's own repo→gate-command table and has no bearing on a hand-applied Workload, which carries its gate command inline via spec.gateProfile instead (see below).

What the draft PR installs: the foreman chart (operator + CRDs: Workload, AgenticTask, Agent, FleetNode, AgentRelease, ModelProfile) and three Agent CRs: coder (role coder, model self-hosted via LiteLLM, Job-mode execution), gate (role verifier, deterministic, with no LLM, runs run_gate_job), reviewer (role reviewer, model openrouter/auto, read-only tools). All three inference through LiteLLM's cloud-proxy provider (spec.providerConfig), pointed at the litellm-key-foreman Secret the LiteLLMVirtualKey operator manages directly, so no LiteLLM key is duplicated into the foreman-agent 1Password-sourced Secret.

Prerequisites the owner must create before the trial can run (not automated by the PR):

• A GitHub App (the trial used foreman-lukeevanstech, ID 4718328: Contents RW / Pull requests RW / Issues R / Metadata R, webhook off, deleted 2026-08-26 together with its 1Password item) installed on the account; its private key lives on the foreman-github 1Password item (Talos vault, fields APP_ID, INSTALLATION_ID, PRIVATE_KEY). An external-secrets GithubAccessToken generator (foreman/app/githubaccesstoken.yaml, now under .archive/) mints hourly installation tokens scoped to the repositories listed there. Add a repository to that list to let foreman work on it.
• The litellm LiteLLMProxy cutover PR must land (or be otherwise verified) before the LiteLLMVirtualKey in this PR can resolve. See that PR's Verification section for what was actually observed against the litellm-operator's validating webhook.

Trial procedure. Apply one Workload by hand against a real issue in a repository the generator's repositories list covers:

apiVersion: foreman.llmkube.dev/v1alpha1
kind: Workload
metadata:
  name: foreman-trial-issue-<N>
  namespace: ai
spec:
  intent: "<one-line description of the fix>"
  repo: <owner>/<repo>
  issues: [<N>]
  coderAgentRef:
    name: coder
  verifierAgentRef:
    name: gate
  reviewerAgentRefs:
    - name: reviewer
  # Only needed for a non-Go repository. An omitted gateProfile resolves to the Go
  # preset (gofmt/govet/golangci-lint/go test), byte-identical to no field at
  # all. See LLMKube's docs/site/foreman/language-gates.md for the built-in
  # presets (python, rust, node, generic) and the `commands` overrides.
  # gateProfile:
  #   language: python

Watch it decompose and run:

kubectl get workload,agentictask,fleetnode -n ai -w

A review GO opens the pull request itself (Workload.spec.openPullRequest defaults to true for an issue-batch Workload), with no dispatch bridge involved. Flux does not own these hand-applied objects, so clean up by name afterward: kubectl delete workload foreman-trial-issue-<N> -n ai, plus any decomposed AgenticTasks that linger as terminal objects (kubectl get agentictask -n ai).

Decision checklist (pick-up point)¶

• hermes: the web chat surface is already live via hermeswebui; the open question is only which bot-token platforms (Discord/Slack/etc.), if any, to enable next (each needs a bot token in 1Password)
• hermes: MCP wiring, resolved by the ToolHive VMCP aggregate (#4594)
• hermes: keep approvals.mode: manual (default) or smart?
• hermes: image tag to pin (jory runs v2026.7.7; newer exists) + digest.
• CephFS: StorageClass name (ceph-filesystem proposed), MDS memory limit, snapshot class now or later?
• Model cache size: 100Gi (proposed) or jory's 200Gi?
• PR C: netpol rules per namespace, or gateway-URL wiring instead?
• seerr MCP: include in PR C or leave skipped?
• PR F: worth doing now, and which model?

Deferred / skipped registry¶

Deferred (right idea, wrong time): ToolHive VirtualMCPServer aggregate (litellm already aggregates + semantically filters; revisit if external agents need one URL: hermes could be that trigger), litellm complexity auto-router + cost economics (need a paid cloud roster), memini-summary dedicated model (GPU pressure; PR F could host it on CPU instead), repo-wiki, speculative decoding, HF-token ExternalSecret, foreman (trialled and archived 2026-08-26; the Foreman section above carries the reinstatement recipe).

Skipped with reasons: openclaw + hermes-parallel runtimes as always-on personas (hermes is being adopted deliberately instead), comfyui + miso-gallery + comfyui-mcp (AMD ROCm hardware), llama-strix / llama-ryzen / llama-vision serving (AMD/Vulkan + DRA on hardware we lack), memory-mcp (fragments memini), litellm HA/public/OIDC/ChatGPT deltas (his conventions, not ours), .agents foreman + pr-review instructions (document infrastructure we don't run; our sorting + add-app files verified better than his), dispatch (pointless without foreman; revisit only together).

toolhive-embed (redundant with all-minilm) above no longer holds: the VMCP tool optimizer's all-minilm embeddings failed on kubectl tool descriptions over MiniLM's 512-token limit, so a CPU qwen3-embedding server (toolhive/embed/) was adopted after all. See docs/docs/architecture/ai-llm-stack.md § "MCP gateway (VirtualMCPServer)" (#4595).