Skip to content

LiteLLM

Purpose

LiteLLM is an OpenAI-compatible LLM gateway in the ai namespace. It fronts the local llmkube (llama.cpp) model-serving tier and cloud providers behind one API, so in-cluster consumers target a single endpoint and get virtual keys, per-key budgets, request logging, model-group routing, fallbacks, and an admin UI. See the AI / LLM stack page for how it fits the wider stack.

Design decisions

  • Deployed as a LiteLLMProxy CR (litellm-operator, applyMode: file) rather than a HelmRelease; the operator owns the Deployment, Service, ConfigMap, and HTTPRoute it renders from that CR plus the adopted LiteLLMModel / LiteLLMMCPServer CRs (proxyRef: litellm).
  • Single replica: the previous Recreate strategy existed to avoid rolling-update races against Prisma schema migrations at startup; the operator's Deployment doesn't expose a strategy field, so a second replica during a rollout is a theoretical (not yet observed) risk worth watching if this is ever scaled up.
  • State lives in the shared CloudNativePG postgres18 cluster (its own litellm database + role). The database/role are provisioned once, out of band, with no init container recreating them on every reconcile, so the app is PVC-less and needs no VolSync.
  • Response cache in the shared Dragonfly over *.svc.cluster.local, with a deliberately short TTL so usage tracking stays accurate.
  • generalSettings.store_model_in_db: false: the LiteLLMModel CRs are the model source of truth, rendered into config.yaml by the operator (applyMode: file). This trades the admin-UI "add a model" workflow for GitOps-reviewable model changes.
  • Prometheus success/failure callbacks enabled for metrics.
  • Auth is the built-in master key (apiAccess.masterKeyRef) + UI credentials (no external SSO for an internal-only service). Existing DB-stored virtual keys (created before this migration) keep working unchanged. They authenticate against the DB, independent of store_model_in_db.
  • Routing is an internal-only HTTPRoute (operator-managed via spec.route) on the envoy-internal listener at litellm.${SECRET_DOMAIN}; the API (/v1/*) and admin UI (/ui) share one port.
  • Secrets come from an ExternalSecret pulling the litellm 1Password item (Talos vault).

Deploy gotchas

  • Prometheus scrape: the /metrics endpoint needs require_auth_for_metrics_endpoint: false, and the ServiceMonitor path must be /metrics/ (trailing slash, with redirect-follow off). Otherwise the scrape silently gets nothing.
  • The liveness/readiness probe path is /health/liveliness (upstream's spelling); the operator defaults readiness to /health/readiness instead, so both probes are set explicitly in litellmproxy.yaml. /v1/health requires auth and runs heavy backend probes, too heavy for kubelet.
  • Never rotate LITELLM_SALT_KEY: rotating it makes DB-stored provider/virtual keys undecipherable (config-file keys are unaffected). Note this on the 1Password item.
  • Keep provider API keys out of Git by referencing them as os.environ/<NAME> in a LiteLLMModel's params.apiKey (the operator wires the env var from a Secret automatically when you use apiKeyRef instead of a literal os.environ/ string).
  • Two GrafanaDashboard CRs: litellm is fetched from grafana.com via URL (Flux postBuild does not process remotely fetched JSON, so no $${var} escaping is needed); litellm-detail is vendored JSON in a configMapGenerator with the kustomize.toolkit.fluxcd.io/substitute: disabled annotation, since that dashboard's JSON has no ${VAR} placeholders needing Flux substitution and the annotation stops Flux blanking any stray ${...} inside it.
  • The dashboard ConfigMap and every LiteLLMProxy/LiteLLMModel/LiteLLMMCPServer CR set metadata.namespace explicitly (Checkov CKV_K8S_21 for the ConfigMap; the CRs are namespaced by convention with the rest of the app).
  • Liveness initialDelaySeconds: 180: the mcp_semantic_tool_filter embeds the whole MCP tool registry synchronously on every start into a LocalIndex that is never persisted. A cold start measured ~17s with the embedding backend idle and 104s when it was busy. The previous 30s + 6x10s probe killed the container at ~90s, before it served a request; because the proxy exits 0 on SIGTERM this never showed as CrashLoopBackOff, only KubePodNotReady, and each kill re-queued the whole registry onto the embedder's two slots, making the next start slower still: 62 restarts in two hours. The LiteLLMProxy CRD has no startupProbe field, so initialDelaySeconds is the only knob; periodSeconds/failureThreshold still apply once the pod is past that window.

Timeouts

request_timeout and TimeoutErrorRetries in litellmproxy.yaml are tuned together:

  • request_timeout: 1500 must stay under the tightest caller budget, repowiki's LLM_TIMEOUT at 1800s.
  • TimeoutErrorRetries: 0 overrides the default num_retries: 2 for timeouts only; every other exception class still gets two retries. Without this override, a timing-out call burned three times request_timeout before failing over, well past the point the caller had already given up, and the retries just re-queued a doomed prompt onto the same two GPU slots that caused the timeout in the first place. Retrying a timeout is close to useless here regardless, since self-hosted has a single deployment (llama-nvidia): a retry re-queues onto the same GPU.

Alerting

prometheusrule.yaml thresholds and windows:

  • Thresholds sit lower than a busier upstream default: with one replica, increase() is the true event count, with no cross-replica summing, so small counts already mean something.
  • LiteLLM creates each counter series at its first event, and increase() never counts that first value, so the Claude alerts that must catch a single event also match a series absent an hour earlier (X unless X offset 1h). .github/scripts/tests/prometheus/litellm.yaml tests those Claude alerts against that series shape and a quiet restart, in lefthook and CI.
  • The *_fallbacks_total counters only appear after the first fallback event (LiteLLM registers them lazily), so LiteLLMFallbackChainExhausted and LiteLLMModelFailover carry no absent() guard: their absence is the healthy steady state, not a stale metric.
  • LiteLLMDeploymentOutage's rate() window (3m) is kept well under its for: (10m) on purpose. A window at or above for: lets one isolated failure extrapolate rate() > 0 across the whole for: period, so the alert would keep firing after the deployment recovered. A shorter window needs failures recurring in every window across the full for: duration.
  • InferenceServiceDown reads up from the llmkube-models ServiceMonitor and aggregates with max by (service), so it only fires once every scrape endpoint of a backend is unreachable. Its 15m for: absorbs pod restarts and cold GGUF loads; a fully-scaled-to-zero pod drops its endpoint entirely and goes stale rather than reporting up == 0, which this alert doesn't cover.
  • The four LiteLLMClaude* alerts watch the claude budget. BudgetLow and SpendFast read the remaining budget and 24h spend. BudgetMetricAbsent covers what BudgetLow cannot: the budget filter sets the remaining-budget gauge on every claude call, so a successful call with no gauge means provider_budget_config is unset. RequestRejected fires on any Anthropic 400 that failed over, since that is always a request param Anthropic does not accept (add it to additional_drop_params in models/claude.yaml). A reply whose model is not claude is the same failure seen from a client.

Adding a cloud provider

litellm/app/models/kustomization.yaml used to carry these as commented-out LiteLLMModel templates from Jory's fleet; they're kept here instead, disabled until wired up so a missing key can't fail the deploy. To enable one: add its key to the litellm 1Password item, add the matching line to externalsecret.yaml's target.template.data, then add the LiteLLMModel CR to litellm/app/models/ and list it in that directory's kustomization.yaml.

apiVersion: litellm.home-operations.com/v1alpha1
kind: LiteLLMModel
metadata:
  name: kimi-k2 # Moonshot
  namespace: ai
spec:
  modelName: kimi-k2
  proxyRef: litellm
  params:
    model: moonshot/kimi-k2
    apiBase: https://api.moonshot.ai/v1
    apiKey: os.environ/MOONSHOT_API_KEY
  info:
    mode: chat
    supportsFunctionCalling: true
---
apiVersion: litellm.home-operations.com/v1alpha1
kind: LiteLLMModel
metadata:
  name: minimax # MiniMax (Anthropic-format)
  namespace: ai
spec:
  modelName: MiniMax
  proxyRef: litellm
  params:
    model: minimax/MiniMax-M2
    apiBase: https://api.minimax.io/anthropic/v1/messages
    apiKey: os.environ/MINIMAX_API_KEY
    additional:
      cache_control_injection_points:
        - { location: message, role: system }
        - { location: message, index: -1 }
  info:
    mode: messages
    supportsFunctionCalling: true
    supportsVision: true
---
apiVersion: litellm.home-operations.com/v1alpha1
kind: LiteLLMModel
metadata:
  name: zen-glm # OpenCode "zen" (one key, many models)
  namespace: ai
spec:
  modelName: zen/glm
  proxyRef: litellm
  params:
    model: openai/glm-4.6
    apiBase: https://opencode.ai/zen/go/v1
    apiKey: os.environ/OPENCODE_API_KEY
  info:
    mode: chat
    supportsFunctionCalling: true

Operational notes

  • Reconcile chain: flux reconcile source git flux-system → external-secrets → onepassword → litellm-operator → the litellm Kustomization → its LiteLLMProxy.
  • kubectl get litellmproxy litellm -n ai shows the Ready condition, observedModels count, and readyReplicas; kubectl describe surfaces render/apply errors (e.g. a model referencing a Secret key that doesn't exist) that never reach the pod at all.
  • API round-trip test:
curl -sS https://litellm.${SECRET_DOMAIN}/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-group>","messages":[{"role":"user","content":"ping"}]}'
  • Add models via a LiteLLMModel CR under litellm/app/models/ (proxyRef: litellm). The operator hashes the rendered config into a pod annotation and rolls the Deployment automatically, with no manual restart or Reloader annotation needed.
  • just kube litellm-spend [days] prints requests, spend, failures and latency per key and model group, plus the provider budgets, from the proxy's spend logs (scripts/litellm_spend.py, run inside the pod so the master key stays there).
  • The Grafana "LiteLLM" / "LiteLLM Detail" dashboards and the Gatus check confirm health and metrics flow.