AI / LLM stack¶

Self-hosted LLM stack in the ai namespace, fronted by LiteLLM. Patterns adapted from joryirving/home-ops, re-targeted to this cluster: NVIDIA L4 GPUs + llama.cpp (llmkube).

Components¶

LiteLLM persists to CNPG postgres18 (litellm db) and caches in Dragonfly. Internal-only route (litellm.${SECRET_DOMAIN} on envoy-internal).

Model serving (llmkube)¶

llmkube is the sole GPU inference tier. Each model is declared as a Model CR (weights source + hardware) plus an InferenceService CR (the serving pod), one file per model under kubernetes/apps/ai/llmkube/models/.

Active model. One GPU model serves both LiteLLM groups:

The model is Qwen3.8-27B Heretic-abliterated (0bserverx RVN Q4_K_S, MTP head retained): picked as the closest-to-vanilla uncensored build (KL ~0.0085 vs base, refusals 0–1/100, official chat template). Its hybrid Gated-DeltaNet layout keeps KV at ~32KB/token (q8_0), so one 24GB L4 serves 128k of context across 2 slots alongside the weights and the vision projector, with no --rope-scaling yarn and no --override-kv. --kv-unified (#4579) pools that context rather than splitting it statically, so one request may use the whole 128k window while the other slot is idle, instead of a fixed 64k-per-slot cap.

Weight files are declared as hf:// URIs pointing to single-file public GGUFs on Hugging Face. llmkube downloads and caches them on the shared CephFS RWX modelCache PVC (ceph-filesystem storage class, 100Gi: ~35Gi of GGUFs today plus headroom), so a cold start auto-heals without manual staging. This PVC replaced per-model RWO ceph-block PVCs plus cURL staging Jobs, which existed only because of the "Talos has no ceph kernel module" misdiagnosis (KB-025); CephFS mounts natively. Weights are re-downloadable, so the cache carries no VolSync.

Anti-affinity (podAntiAffinity) keeps one resident model per L4: the cluster has 3 cards and runs 1 model, preserving two cards for other GPU workloads (Plex/Jellyfin transcodes, Whisper). No model swapping occurs during normal operation. The gpu-preemptible PriorityClass is set on all llmkube pods so higher-priority workloads can evict them if needed.

The model is vision-enabled: the InferenceService mounts the repo's mmproj-*-Q8_0.gguf multimodal projector alongside the main GGUF. This is the model that loupe (image analysis) consumes via LiteLLM.

To add a model: drop a Model + InferenceService manifest under llmkube/models/, add a LiteLLMModel CR under litellm/app/models/, and commit. Flux reconciles both.

Resource budget on the L4¶

The 24GB L4 has ~22.4Gi usable, and llama-nvidia's budget is tight: 14.9Gi weights (Q4_K_S + the embedded MTP head, +0.42Gi) plus 0.6Gi for the mmproj projector (kept on the GPU, see "Vision projector placement" below) plus 3Gi KV (128k context, K at q8_0 and V at q4_0; only the 16 of 64 layers that are full-attention hold KV) plus compute buffers. With MTP on and both KV tensors at q8_0 the card sat at 22.0-22.5Gi and the first real prompt crashed llama-server with "CUDA error: out of memory" in ggml_cuda_pool_vmm::alloc: the mul_mat compute pool grows at decode time, so ~0.5Gi of idle headroom isn't enough. The floor kept free is ≥1.5Gi. Dropping the V-cache to q4_0 bought back 1Gi (K stays q8_0, the recall-sensitive half), and uBatchSize: 512 trims the compute buffers further, at a modest prompt-processing cost. Idle free VRAM measured ~1.8Gi with the projector on the GPU and --kv-unified active (2026-08-25).

Speculative decoding (--spec-type draft-mtp) drafts with that embedded MTP head and verifies with the main model: lossless, same output distribution, it only changes tokens/step. Baseline decode without it measured 15.2 t/s on the L4; expect roughly +30-60% at the typical ~0.75 acceptance rate. --spec-draft-n-max 2 is the consumer-GPU sweet spot from the qwen38-mtp recipe (tunable 2-4); the gain shrinks as concurrent slots rise, which is fine at 2 slots.

Vision projector placement¶

The mmproj projector stays on the GPU, unlike Jory's --no-mmproj-offload (added in #4579 to free ~0.6Gi of VRAM by moving it to host RAM). A post-merge measurement on 2026-08-25 found that trade-off wrong for this cluster: three vision requests against a 64x64 image took 59.7s / 53.8s / 53.3s off-GPU (correct answers, all CPU-bound on the 1024-token image encode) versus low single-digit seconds with the projector on the GPU. loupe depends on this vision path, so latency wins over the 0.6Gi. --image-min-tokens 1024 (Jory's value) keeps vision quality regardless of where the projector runs.

Rollout and staging safety¶

• rolloutPolicy.waitForIdle defers a pod-template rollout (a Renovate image bump, for example) until llama.cpp's own /slots endpoint reports both slots idle, instead of killing an in-flight agent session mid-response. It falls back to proceeding after idleTimeoutSeconds (86400s) so a genuinely stuck slot can't block upgrades forever.
• Never set skipModelInit. It looks safe, since the controller already stages the GGUF and mmproj into the shared cache PVC and the init container would just log "already cached", but on llmkube 0.9.x the flag also blanks the rendered --model argument, because that argument is derived from the init container's staging step. llama.cpp then starts in router mode with zero models: the pod goes 1/1 Ready and every request fails with "model not found". This was set and reverted once already (#3792).
• No runAsNonRoot: the operator's model-cache-prep init container needs scoped root (drop ALL, add CHOWN + FOWNER only, no privilege escalation, read-only rootfs) to chown the shared cache on fsGroupPolicy: None backends like CephFS. This is llmkube's documented upstream compatibility path. Runtime containers still run as uid 1000.
• The image needs a llama.cpp build from release week 2026-08-14 or later; older builds don't register the qwen35 architecture and refuse to load the GGUF.

Grafana dashboards¶

Two dashboards are ported from joryirving/home-ops, llama-server and llm-nvidia-node (the latter correlates DCGM GPU telemetry with llama.cpp's own throughput metrics so a slow request can be read against GPU or host pressure instead of guessed at). Both escape every Grafana template variable by doubling its leading $ ($${datasource}, $$service, $$node, $$instance, $$__range, $$__all) so Flux postBuild passes the literal ${...}/$... through to Grafana instead of blanking it, and both resolve the datasource variable to the lowercase prometheus datasource (KB-021).

• llama-server: the upstream "Prompt cache (from logs)" panels were dropped, since they query a victoriametrics-logs-datasource this cluster's Grafana doesn't have wired (only prometheus/alertmanager GrafanaDatasources exist).
• llm-nvidia-node: three deviations from the source, each verified live against this cluster's kube-prometheus-stack rather than assumed from the source repository:
• namespace="llm" becomes namespace="ai"; Jory's cluster names the LLM namespace llm.
• DCGM_FI_DEV_FB_USED's pod-level panels ("Largest pod VRAM", "VRAM by pod") and the $$namespace/$$pod variables key off exported_namespace/exported_pod instead of namespace/pod. This cluster's dcgm-exporter ServiceMonitor keeps namespace/pod as the exporter's own scrape-target identity (an honor_labels collision) and carries the real workload mapping under exported_namespace/exported_pod, confirmed by querying DCGM_FI_DEV_FB_USED directly: the raw namespace/pod labels read gpu-operator/nvidia-dcgm-exporter-xxxxx (the exporter itself), while exported_namespace/exported_pod correctly read ai/llama-nvidia-....
• The $$instance variable populates from DCGM's hostname label instead of its instance (scrape-endpoint IP:port) label, and every DCGM_FI_DEV_* query filters on hostname=~"$$instance" accordingly. DCGM's hostname label is byte-identical to node-exporter's node_uname_info nodename label (both report the node's short hostname), which the Host CPU/RAM panels already join against; with the raw per-pod instance address, $$instance could never match nodename and those two panels rendered empty in the upstream JSON.

Model groups¶

LiteLLM model_name groups make the serving tier transparent to clients:

• self-hosted: llama-nvidia (order 1), with live cloud fallbacks via router_settings.fallbacks: claude first, then openrouter/auto (both keys are in the litellm ExternalSecret).
• claude: Claude Sonnet 5.5 on the Anthropic API, paid from a monthly provider credit, and the default model for most in-cluster callers. router_settings.provider_budget_config caps anthropic spend at that credit per 30 days, after which requests fall back to self-hosted (then openrouter/auto). LiteLLMClaudeBudgetLow and LiteLLMClaudeSpendFast alert on the remaining budget and the daily spend rate. context_window_fallbacks also sends prompts too large for self-hosted here. Spend is computed from the per-token prices in models/claude.yaml, because this LiteLLM build has no price entry for the model; if those are missing the budget never trips. The LiteLLM window starts at first spend rather than the credit's renewal, so the hard stop is the workspace spend limit set in the provider console. repowiki stays on self-hosted: its overnight volume would exceed the credit. A key's models list gates only the model a caller requests, not router fallbacks (tested 2026-10-18: a self-hosted-only key still fell back to claude), so the budget, not key scope, is what bounds fallback spend. The key (ANTHROPIC_API_KEY in the litellm item) has an expiry date recorded on that item, and an expired key falls back to self-hosted silently.
• self-hosted-uncensored: the same llama-nvidia backend under a second name. Kept as its own group because its fallback semantics differ: no cloud fallback by design (a cloud model would reintroduce refusals), and existing clients keep working unchanged.

In-cluster consumers¶

The in-cluster apps below, all outside the ai namespace, route through LiteLLM at http://litellm.ai.svc.cluster.local:4000/v1 and, except suggestarr, the claude model, falling back to self-hosted and then openrouter/auto. The model column is where to change it:

Every other key is used inside ai or by hand, reading its own litellm-key-<name> Secret. Scope does not stop router fallbacks, so a self-hosted-only key can still reach claude:

.github/scripts/check_virtualkey_docs.py fails when a LiteLLMVirtualKey has no row in this section, so a new key's spend is never missing from it.

The seven apps in the first table live outside the ai namespace, so none of them mounts an ai Secret directly. Each gets an operator-issued scoped key instead of the account-wide master key: a LiteLLMVirtualKey CR in litellm/app/virtualkeys/<app>.yaml (ai namespace) mints the key and writes it to an in-namespace Secret; a paired PushSecret then writes that key back to the litellm 1Password item as property LITELLM_<APP>_API_KEY (updatePolicy: Replace, refreshInterval: 1h). The consumer's own ExternalSecret extracts that property like any other litellm-item field, with no cross-namespace Secret access required. See the consumer's own externalsecret.yaml for the exact template line. suggestarr is the exception: it stores AI credentials in its own config, so its key is entered once in the app's AI settings from that 1Password property and must be re-entered there if the key is ever re-minted.

Rollout (staged)¶

1. LiteLLM uplift: model-groups + router fallbacks + commented hooks for MCP / embeddings / cloud providers.
2. ToolHive + MCP: operator + curated MCP servers (kubectl, flux, talos, searxng) wired into LiteLLM mcp_servers.
3. memini: agent memory; embeddings + rerank via tiny CPU llama.cpp servers, consolidation via LiteLLM.
4. llmkube: operator + CephFS modelCache; 2 active models at the time (self-hosted, self-hosted-uncensored, since consolidated onto one backend, see "Active model" above).
5. Ollama decommission: Ollama removed; contracthound/subspy/loupe repointed to LiteLLM.

Each layer is a separate commit on one branch (one PR). mcp_servers and mcp_semantic_tool_filter are now fully active (rendered by litellm-operator from the LiteLLMMCPServer CRs and litellmSettings in litellm/app/litellmproxy.yaml), and one cloud provider is live: openrouter/auto is an active LiteLLMModel serving as the self-hosted group's fallback. The remaining cloud-provider stubs are commented reference in litellm/app/models/kustomization.yaml.

Since ported from Jory's repository: hermes (plus a hermeswebui chat frontend). Still not ported: openclaw (agent runtime) and agentmemory (memini covers agent memory here).

How to extend LiteLLM¶

The gateway is litellm-operator-managed: a LiteLLMProxy CR (litellm/app/litellmproxy.yaml, applyMode: file) plus LiteLLMModel and LiteLLMMCPServer CRs the operator adopts via proxyRef: litellm and renders into config.yaml, rolling the Deployment on change. There is no litellm/app/configmap.yaml any more. store_model_in_db: false in the proxy's generalSettings makes the rendered CRs, not the Postgres-backed admin UI, the model source of truth.

• Add a backend to a group: add a LiteLLMModel CR under litellm/app/models/ with an existing spec.modelName and the next params.additional.order. LiteLLM balances / fails over within the group.
• Add a cloud provider: add the key to the litellm 1Password item, add a line to externalsecret.yaml's target.template.data, then add a LiteLLMModel CR under litellm/app/models/ (ready-made templates for Jory's set are in LiteLLM: Adding a cloud provider). Don't reference an os.environ/KEY that isn't in the secret. The pod env read fails at startup.
• Fallbacks: litellmproxy.yaml's routerSettings.fallbacks is a list of {model_name: [fallback, …]}.
• A consumer's scoped key (same namespace): add a LiteLLMVirtualKey CR in the consumer's own app directory (proxyRef: litellm, secretName: litellm-key-<app>, secretKey: api-key) rather than handing out the master key. The operator creates the key via the proxy's admin API and writes it to that Secret; consume it with a secretKeyRef. See memini/app/virtualkey.yaml.
• A consumer's scoped key (cross-namespace): a consumer outside ai can't mount an ai Secret directly. Add the LiteLLMVirtualKey CR (ai namespace) plus a paired PushSecret under litellm/app/virtualkeys/<app>.yaml instead. The PushSecret writes the operator-minted key back to the litellm 1Password item as LITELLM_<APP>_API_KEY, and the consumer's existing ExternalSecret extracts it like any other field on that item. See "In-cluster consumers" above.

Consuming the stack from a workstation (opencode)¶

Any OpenAI-compatible client can drive the self-hosted models through the gateway: internal route https://litellm.${SECRET_DOMAIN}/v1, models self-hosted and self-hosted-uncensored. opencode is wired this way as a custom provider:

• Provider: @ai-sdk/openai-compatible, baseURL: https://litellm.${SECRET_DOMAIN}/v1. Put the LiteLLM key in the client's own credential store (opencode: opencode auth login → ~/.local/share/opencode/auth.json), never as a literal in a shared or committed config.
• MCP tools: point a remote MCP server at LiteLLM's MCP gateway, https://litellm.${SECRET_DOMAIN}/mcp/ (the litellm-mcp-server), with the LiteLLM key as Authorization: Bearer …. Curate the server set with the x-mcp-servers header (e.g. kubectl,flux,talos,searxng). Requesting all servers times out, and the full tool list bloats every request (heavy on the small-context local models, so prefer a frontier model for tool-heavy work).
• Gotchas: the self-hosted model is a Qwen thinking model (send think: false / enable_thinking: false, or lower reasoning_effort, for non-reasoning output, since Qwen3.8 defaults to heavy reasoning). Both group names hit the same backend, so there is no model-swap cost for switching between them. See the context-budget note below for opencode's window requirements.

Prefer a scoped LiteLLM virtual key (/key/generate, limited to the self-hosted* models) over the master key for any workstation client: it's revocable on its own.

Workstation Claude Code reaches the talos MCP through litellm/app/virtualkeys/claudecode.yaml: a LiteLLMTeam whose only MCP server is talos and only model self-hosted-uncensored (no cloud fallback), plus a key in that team mirrored to the litellm item as LITELLM_CLAUDECODE_API_KEY. The client calls /mcp/ with an x-mcp-servers: talos header.

opencode and the context budget¶

opencode's agent sends ~41k tokens before any user input: its system prompt plus built-in tool schemas (measured; LiteLLM is not injecting MCP tools, a plain request is ~18 tokens and one with a tool is ~130). Qwen3.8-27B's 262k native context makes this a non-issue: --kv-unified (#4579) lets one request use the full 131072-token window (contextSize in llmkube/models/qwen3.8-27b.yaml) while the other of the 2 parallelSlots sits idle, rather than splitting it into a static 64k-per-slot cap. Either group hosts opencode; use self-hosted-uncensored if a cloud fallback mid-session would be unwelcome.

The --rope-scaling yarn + --override-kv workaround the previous Qwen3-30B-A3B uncensored model needed (the pinned server hard-capped slots to the trained context even with correct yarn scaling arguments, llama.cpp#22140) is retired with it. The verification habit it taught still stands: a Ready phase and a clean kustomize build do not prove the served window. Confirm with /props (.default_generation_settings.n_ctx == 131072, the full unified contextSize, not a per-slot share) after any context change.

MCP tools (ToolHive)¶

Layer 2 runs the StackLok ToolHive operator (separate CRDs + operator charts, see toolhive/app/ocirepository.yaml for the current pin) in ai, an MCPGroup (mcp-tools), and these MCP servers, all wired into LiteLLM's mcp_servers:

kubectl + flux share read-only access built from this cluster's API groups with core secrets omitted, split across two ClusterRoles bound to the same ServiceAccount (plus a second SA for flux-mcp; kubectl-mcp-readonly for the aggregation, kubectl-mcp-readonly-explicit for everything the aggregation doesn't cover). The split is mandatory, not a style choice: a ClusterRole carrying an aggregationRule has its rules field owned by the aggregation controller, which overwrites anything written alongside it, so any rules placed on that role are dead on arrival. kubectl-mcp-readonly mirrors the built-in view ClusterRole's selector and expands automatically as operators ship aggregate-to-view ClusterRoles; the explicit role lists the core group by name (secrets omitted) and this cluster's other API groups, minus auth/* and onepassword.com, as resources: ["*"], which is safe there because Secret objects only exist in the core group. Keep the non-core group list in sync with kubectl api-resources as you add CRDs. The talos MCP mounts a talos.dev ServiceAccount-minted os:reader talosconfig.

The mcp_semantic_tool_filter is on (top_k 8, embeddings via the qwen3-embedding model on the CPU toolhiveembed pod): with 9 servers' worth of tools it trims each request to the most relevant ones. It ran on all-minilm until that model's 512-token cap was found to be failing the filter's own 625-token registry payload on every startup, leaving it silently inactive. github + grafana are read-only, via a fine-grained PAT (toolhive-github) and a Grafana Viewer service-account token (toolhive-grafana).

MCP gateway (VirtualMCPServer)¶

A VirtualMCPServer named mcp-gateway (toolhive/gateway/virtualmcpserver.yaml) aggregates every backend in the mcp-tools MCPGroup behind one endpoint, for hermes, workstation opencode, and any other client that talks MCP directly instead of going through LiteLLM's mcp_servers wiring. Tool names are namespaced {workload}_<tool> (e.g. searxng_search) to resolve collisions across backends. Because 9 backends' worth of raw tool definitions is too large for most client context windows, the built-in optimizer exposes only find_tool/call_tool to clients and resolves the right backend tool semantically. It embeds via qwen3-embedding (Qwen3-Embedding-0.6B, 8k context; toolhive/embed/), a dedicated CPU llama.cpp server, rather than LiteLLM's all-minilm (all-MiniLM-L6-v2): MiniLM's BERT architecture has a hard 512-token limit and some kubectl tool descriptions exceed it, which used to terminate every VMCP session with an OpenAI 400 "input larger than max context size" error. The same cap turned out to be breaking LiteLLM's own mcp_semantic_tool_filter above, which now embeds here too; all-minilm stays in place for memini alone.

• External URL: https://mcp.${SECRET_DOMAIN}/mcp, header x-api-key: <key>. Keys live as fields on the toolhive 1Password item; the workstation's key is TOOLHIVE_WORKSTATION_API_KEY. Add a client by adding a field to that item plus a matching template key in toolhive/gateway/externalsecret.yaml's mcp-gateway-api-keys ExternalSecret. Any key's value in the resulting Secret is accepted (Envoy Gateway's SecurityPolicy doesn't distinguish which one matched), and the SecurityPolicy strips the header before it reaches the vmcp backend, so the key is never forwarded past the gateway.
• In-cluster URL: http://vmcp-mcp-gateway.ai.svc.cluster.local:4483/mcp, no auth (anonymous incomingAuth, because the API-key gate lives at the Envoy Gateway edge, not in the vmcp app itself). The ToolHive operator names a VirtualMCPServer's Service vmcp-<name>, distinct from an MCPServer's mcp-<name>-proxy pattern (see "Adding an MCP server" below).
• The gateway's own /health is unauthenticated, but the SecurityPolicy requires x-api-key on every path the HTTPRoute matches, /health included, so an unauthenticated Gatus probe would get 401. httproute.yaml disables Gatus auto-monitoring for this route rather than carving out an unauthenticated path just for the health check.
• The optimizer's embedding call to litellm.ai.svc.cluster.local uses a scoped LiteLLMVirtualKey (toolhive/gateway/virtualkey.yaml, models all-minilm + qwen3-embedding), and embeddingServiceTimeout: 600s: a fresh session embeds every tool description through the CPU embedder, measured at ~2 minutes for the full set, so the 60s value from Jory's GPU-sized deployment timed out and stacked retries.
• Sessions are stored in Dragonfly (sessionStorage.provider: redis) so the Deployment can scale beyond one replica without pinning MCP clients to a specific pod.
• Metrics are scraped from the same port via the existing prometheus MCPTelemetryConfig; a Grafana dashboard is imported from ToolHive's upstream OTEL-scrape dashboard JSON. The upstream JSON hardcodes the datasource uid prometheus directly rather than a ${DS_...} placeholder, so there is no __inputs block for grafana-operator's substitution to target; if panels show "datasource not found", this cluster's Prometheus datasource UID doesn't match that literal.

Metrics scraping¶

Both PodMonitors in this layer (toolhive/app/podmonitor.yaml for the per-MCPServer proxy pods, toolhive/gateway/podmonitor.yaml for the vmcp gateway pod) share two traps:

• Neither carries a release: kube-prometheus-stack label; this cluster's Prometheus selects monitors without it, so copying that selector from an upstream example would silently drop the target.
• The ToolHive operator names the container port http, not a numeric port. Prometheus matches podMetricsEndpoints[].port by name, so a numeric value resolves no target at all. /metrics is served on that same port once a telemetryConfigRef is set (MCPTelemetryConfig, created by the toolhive Kustomization). The gateway PodMonitor's label selector (app.kubernetes.io/name=virtualmcpserver, app.kubernetes.io/instance=<CR name>) is verified against the operator's labelsForVirtualMCPServer source, not assumed from an example.

Embedding server tuning¶

The dedicated CPU embedder (toolhive/embed/helmrelease.yaml, toolhiveembed) runs Qwen3-Embedding-0.6B for both the VMCP tool optimizer and LiteLLM's mcp_semantic_tool_filter. Its llama.cpp flags were tuned against several OOM and timeout incidents, all on 2026-08-25:

• --ctx-size 8192 is the total KV budget, split evenly across --parallel slots. At 4 slots that left 2048 tokens per input, and a batched tool-description upsert 400'd (exceed_context_size_error). Two slots give 4096 tokens per input, comfortably above the longest MCP tool description seen (~650 tokens), while keeping the 0.6B model's KV inside the 2Gi memory limit.
• --ubatch-size must be at least the longest single input, since llama.cpp cannot split one pooled-embedding sequence across micro-batches, and it sizes the attention scratch buffers quadratically: 8192 OOM-killed the pod at 2Gi on the first multi-input request. 2048 is ample for the ~650-token descriptions; --batch-size 4096 matches the slot context.
• The optimizer embeds every tool description (~540 inputs, ~100k tokens) when a session starts. At 4 threads that took minutes and the optimizer's request timed out, then retried while the first batch was still queued. Nodes have 24 cores, so this runs --threads 12 with a matching CPU request.
• llama-server's RAM prompt cache (--cache-ram, default 8192 MiB) keeps every processed sequence's KV so later prompts sharing a prefix can reuse it, but embedding inputs never share prefixes, so the cache only grows: it OOM-killed the pod at 3Gi after ~180 tool-description inputs. --cache-ram 0 plus --no-cache-prompt disables it entirely, but RSS still grows ~2Gi transiently while working through the optimizer's ~540-input batch and settles back to ~1Gi after; 3Gi OOM-killed the pod mid-batch three more times even with caching off, hence the 6Gi memory limit.
• --mmap was removed upstream in favor of --load-mode, and passing the old flag makes the server exit with "invalid argument: --mmap" before it binds a port. --load-mode works on both the old and new builds, so it's used here.

flux-mcp write access (enabled)¶

The flux MCP has had write access to Flux CRDs since this was enabled: flux-mcp-write, a ClusterRole + ClusterRoleBinding in toolhive/mcp-servers/flux/rbac.yaml, grants the flux-mcp ServiceAccount create/patch/update/delete on every Flux kind installed on this cluster, one rule per apiGroup (fluxcd.controlplane.io, helm.toolkit.fluxcd.io, kustomize.toolkit.fluxcd.io, notification.toolkit.fluxcd.io, source.toolkit.fluxcd.io), each listing its resources by name rather than resources: ["*"], because a wildcard trips Trivy KSV-0046 and Checkov's wildcard-RBAC check even when the apiGroups are this narrow. The resource list for each apiGroup comes from kubectl api-resources --api-group=<group> -o name against the live cluster, and deliberately excludes subresources like <kind>/status: flux-operator-mcp's write tools only ever patch spec or annotations on the main object, never status. Nothing outside Flux CRDs, and no secrets access (core "" is never included). image.toolkit.fluxcd.io has no rule because this cluster doesn't run the Flux image-automation controller. Add one (imagepolicies, imagerepositories, imageupdateautomations) if that ever changes, and add any other new Flux kind by hand when a component upgrade introduces one. This lets it (and therefore any model behind LiteLLM) reconcile, suspend, resume, apply, and delete Flux objects.

The flux-operator-mcp image's own --read-only flag defaults to false and this deployment has never set it, so the write tools were already registered at the MCP layer before this change. RBAC was, and remains, the only enforcement boundary.

This grants an LLM mutate access to the cluster's GitOps controller; the owner has accepted that blast radius. To revoke, delete the flux-mcp-write ClusterRole and ClusterRoleBinding from toolhive/mcp-servers/flux/rbac.yaml (the two read-only bindings are unaffected).

Adding an MCP server¶

Drop an MCPServer (operator-managed) or MCPServerEntry (remote URL) with groupRef: mcp-tools under toolhive/mcp-servers/<name>/, list it in that dir's kustomization.yaml, then add its endpoint to LiteLLM's mcp_servers. The service name depends on the transport:

• Native streamable-http transport (e.g. kubectl, flux): ToolHive creates mcp-<name> on the spec's mcpPort.
• stdio transport with proxyMode: streamable-http (e.g. github, grafana): ToolHive creates mcp-<name>-proxy on the spec's proxyPort (typically 8080).

Registering an existing app's Service as an MCPServerEntry (arrmcp, searxng) hits one more trap: ToolHive rejects a remoteUrl host containing cluster.local unless allowPrivateEndpoint is set. The short svc.ns form resolves the same address and is accepted as-is.

What clients actually see¶

The gateway runs ToolHive's semantic optimizer, so a client that lists tools gets exactly two meta-tools, not the ~540 aggregated ones: find_tool(tool_description) returns the backend tools that best match a natural-language need (embedded via qwen3-embedding), and call_tool invokes one by its <backend>_<tool> name. Point agents at those two; do not expect kubectl_* names in tools/list.

Warm-up: the first session after the gateway or embedder pod restarts builds the embedding store (~540 descriptions through the CPU toolhiveembed server, ~1–2 min) and that first initialize can drop with "connection closed"; the store persists, so every later session initializes in about a second. If sessions keep failing, check toolhiveembed for OOMKilled (its RSS peaks ~3.2Gi during the build; limit is 6Gi) and the VMCP logs for exceed_context_size_error.

Agent memory (memini)¶

Layer 3 runs memini (SQLite backend) for agent long-term memory, plus two tiny CPU llama.cpp model servers in ai:

• llama-embed: all-MiniLM-L6-v2 (384-dim), --embeddings, OpenAI /v1.
• llama-rerank: Qwen3-Reranker-0.6B, --rerank.

Both run on CPU (ghcr.io/ggml-org/llama.cpp:server, GPU/Vulkan bits stripped). The L4s are spoken for by llmkube, and these models are small (~30 MB / ~600 MB). memini's consolidation LLM is LiteLLM's self-hosted group.

Secrets: a generated MEMINI_API_KEY (Talos vault item memini) + a scoped LiteLLMVirtualKey (litellm-key-memini, see memini/app/virtualkey.yaml) rather than the master key. Data PVC via the volsync component (10Gi). Route: memini.${SECRET_DOMAIN} (envoy-internal).

To move embeddings onto the GPU later, swap llama-embed/llama-rerank for llmkube InferenceServices and repoint MEMINI_EMBED_BASE_URL / MEMINI_RERANK.

repowiki¶

repowiki generates an AI-written wiki for every repository in the LukeEvansTech GitHub account and serves it as a static site. Two controllers share one RWO ceph-block PVC:

• mkdocs (Deployment): squidfunk/mkdocs-material in serve --dirty mode, rendering whatever is committed to the Git repository on the shared PVC. This is what the repowiki.${SECRET_DOMAIN} route (envoy-internal) serves.
• repowiki-gen (CronJob, every 12h): clones/updates each repository from repos.txt, has the self-hosted LiteLLM model plan a page set per repository, writes the pages, and commits them into the Git repository on the PVC. A podAffinity pins it to the same node as the mkdocs pod, since the PVC is ReadWriteOnce.

Ported from Jory's repo-wiki; generate.py (in kubernetes/apps/ai/repowiki/app/configmap.yaml) is otherwise unmodified from upstream. Only the Git commit identity and env values changed for this cluster.

Adding or removing a repository: edit repos.txt in configmap.yaml and commit, with no Helm/Kustomize change needed. The generator only regenerates a repository once its HEAD SHA changes since the last successful run, so a new entry is picked up on the CronJob's next occurrence.

Pacing: MAX_REPOS_PER_RUN: "2" caps each run to two repositories (whichever are most stale), so a full pass over a large repos.txt takes several days, not one run. This is deliberate. MAX_PAGES_PER_REPO: "20" pages per repository through an LLM planning + writing pass is not cheap, and the CronJob shares the self-hosted LiteLLM model with every other consumer. Increase MAX_REPOS_PER_RUN only if the model has headroom.

Gotchas¶

• Public repository: no LAN IPs / internal hostnames in Git (see CLAUDE.md). Cluster service DNS and ${SECRET_*} placeholders are fine.
• Metrics: require_auth_for_metrics_endpoint: false and ServiceMonitor path /metrics/ (trailing slash, no redirect-follow) are both required for in-cluster Prometheus scraping.
• CephFS dependency: llmkube's shared modelCache PVC requires ceph-filesystem (RWX). Without it, multi-replica InferenceService pods fail to schedule (only one pod can hold an RWO volume at a time). The ceph-filesystem storage class is provisioned by Rook-Ceph.
• Config changes roll the pod automatically: litellm-operator hashes the rendered config.yaml into a litellm.home-operations.com/config-hash pod annotation, so editing a LiteLLMModel/LiteLLMMCPServer/LiteLLMProxy CR rolls the Deployment on its own, with no Reloader annotation or manual restart needed.
• open-webui's database is still SQLite: on a ReadWriteOnce ceph-block PVC, with strategy: Recreate (see Open WebUI). Web search, RAG embeddings, and Dragonfly-backed websockets are independent of this. Postgres migration is a separate, not-yet-scheduled decision.
• Cross-namespace netpol: kubernetes/apps/ai/netpol.yaml allows ingress to the ai namespace from the network namespace (gateway), plus a second CiliumNetworkPolicy (allow-litellm-from-consumers) that grants the default and custom namespaces ingress to the litellm endpoint specifically. A new consumer namespace needs adding to that policy's fromEndpoints list before its calls to litellm.ai.svc.cluster.local:4000 will connect.
• drop_params is deliberately off: with it enabled, LiteLLM silently strips reasoning_effort before a request reaches the llama.cpp backend (treated as unsupported for the openai/ provider entry), which defeats disabling Qwen3.8's thinking mode (verified 2026-08-25). If a real UnsupportedParamsError 400 ever shows up, scope the drop with a params.dropParams on that model's LiteLLMModel CR instead of re-enabling this globally. Turning drop_params off alone did not forward the param either, because LiteLLM's UnsupportedParamsError check runs independently of it, so the self-hosted and self-hosted-uncensored LiteLLMModel CRs also list reasoning_effort in params.additional.allowed_openai_params (rendered into litellm_params.allowed_openai_params), which is the actual forwarding mechanism.