Skip to content

Bootstrap Process

This document describes the bootstrap process for the Talos Kubernetes cluster using the just task runner.

Overview

The bootstrap process uses a modular, stage-based approach powered by just, a modern command runner. This replaces the previous Bash script-based bootstrap system with a more maintainable and debuggable solution.

Architecture

Task Runner Structure

.justfile                    # Root task runner with common utilities
└── bootstrap/mod.just       # Bootstrap-specific tasks

Helmfile Structure

bootstrap/
├── helmfile.d/             # Modular helmfile structure
│   ├── 00-crds.yaml        # CRD extraction from Helm charts
│   ├── 01-apps.yaml        # Bootstrap applications
│   └── templates/
│       └── values.yaml.gotmpl  # DRY values template
├── kustomize/              # Kustomize-based bootstrap secrets
│   └── apps/
│       ├── external-secrets/   # 1Password Connect credentials
│       └── network/            # Cloudflare tunnel secret
└── mod.just                # Bootstrap task definitions

Prerequisites

Required Tools

All tools are automatically installed via mise:

mise install

Core Tools:

  • just - Task runner
  • kubectl - Kubernetes CLI
  • helm - Helm package manager
  • helmfile - Declarative Helm deployment
  • talosctl - Talos Linux CLI
  • kustomize - Kubernetes configuration management
  • yq - YAML processor
  • op - 1Password CLI (for secret injection; install separately, e.g. via Homebrew, not in the mise toolchain)

Additional Tools:

  • gum - Beautiful shell logging
  • minijinja-cli - Jinja2 template rendering

Required Files

  • talos/clusterconfig/talosconfig - Talos configuration
  • talos/clusterconfig/talos-cluster-*.yaml - Per-node Talos configs

1Password Authentication

The bootstrap process uses 1Password CLI for secret injection. Ensure you're authenticated:

op whoami

If not authenticated:

eval $(op signin)

Bootstrap Stages

The bootstrap process consists of 8 sequential stages:

Stage 1: Talos OS Installation

Command: just bootstrap talos

Applies Talos configuration to all nodes in the cluster.

What it does:

  • Iterates through all nodes from talosctl config
  • Applies Talos configuration to each node
  • Skips nodes that are already configured
  • Uses --insecure flag for initial bootstrap

Example output:

2025-11-04T15:00:00Z INFO Running stage... stage=talos
2025-11-04T15:00:05Z INFO Talos already configured, skipping apply of config stage=talos node=10.1.1.11

Stage 2: Kubernetes Bootstrap

Command: just bootstrap kube

Initializes the Kubernetes control plane.

What it does:

  • Runs talosctl bootstrap on the first controller
  • Retries every 5 seconds until successful
  • Completes when etcd cluster is formed

Example output:

2025-11-04T15:01:00Z INFO Running stage... stage=kube
2025-11-04T15:01:05Z INFO Kubernetes bootstrap in progress. Retrying in 5 seconds... stage=kube

Stage 3: Kubeconfig Retrieval

Command: just bootstrap kubeconfig [lb]

Downloads cluster credentials and configures kubectl.

Parameters:

  • lb - Load balancer type (default: cilium)
  • cilium - Use Cilium LoadBalancer IP
  • node - Connect directly to control plane node

What it does:

  • Fetches kubeconfig from Talos
  • Sets context name to main
  • Optionally updates cluster server address

Example output:

2025-11-04T15:02:00Z INFO Running stage... stage=kubeconfig

Stage 4: Node Readiness Wait

Command: just bootstrap wait

Waits for cluster nodes to be available.

What it does:

  • Checks if nodes are already Ready=True (skip wait)
  • Otherwise waits for nodes to transition to Ready=False
  • Indicates nodes are booting and preparing for configuration

Example output:

2025-11-04T15:03:00Z INFO Running stage... stage=wait
2025-11-04T15:03:05Z INFO Nodes not available, waiting for nodes to be available. Retrying in 5 seconds... stage=wait

Stage 5: Namespace Creation

Command: just bootstrap namespaces

Creates Kubernetes namespaces for all applications.

What it does:

  • Scans kubernetes/apps/*/ directories
  • Extracts Namespace resources from each kustomization
  • Applies namespaces server-side

Example output:

2025-11-04T15:04:00Z INFO Running stage... stage=namespaces

Stage 6: Bootstrap Resources

Command: just bootstrap resources

Deploys critical bootstrap secrets and resources.

What it does:

  • Runs kustomize build bootstrap/kustomize/apps
  • Pipes through just template (minijinja-cli + vals eval) to resolve ref+op:// refs
  • Applies resources server-side

Resources deployed:

  • 1Password Connect credentials (external-secrets namespace)
  • Cloudflare tunnel ID (network namespace)

Example output:

2025-11-04T15:05:00Z INFO Running stage... stage=resources

Stage 7: CRD Installation

Command: just bootstrap crds

Installs Custom Resource Definitions from Helm charts.

What it does:

  • Templates helmfile.d/00-crds.yaml
  • Extracts CRDs using yq post-renderer
  • Applies CRDs server-side

Current CRDs:

  • cloudflare-dns (external-dns chart, network namespace)
  • envoy-gateway (network namespace)
  • grafana-operator (observability namespace)
  • kube-prometheus-stack (observability namespace)

Example output:

2025-11-04T15:06:00Z INFO Running stage... stage=crds

Stage 8: Application Deployment

Command: just bootstrap apps

Deploys bootstrap applications via Helmfile.

What it does:

  • Syncs helmfile.d/01-apps.yaml
  • Deploys applications in dependency order
  • Waits for jobs and pods to be ready
  • Executes post-sync hooks

Applications deployed (in order):

  1. Cilium - Network CNI
    • Post-hook: Waits for Cilium CRDs (kubectl wait --for=create --timeout=2m)
  2. CoreDNS - DNS server
    • Depends on: Cilium
  3. Spegel - OCI registry mirror
    • Depends on: CoreDNS
  4. Cert-Manager - Certificate management
    • Depends on: Spegel
  5. External Secrets - Secret management
    • Depends on: Cert-Manager
  6. OnePassword Connect - 1Password secrets backend
    • Depends on: External Secrets
    • Post-hook: Waits for ClusterSecretStore CRD (kubectl wait --for=create --timeout=2m)
  7. Flux Operator - GitOps operator
    • Depends on: OnePassword Connect
  8. Flux Instance - GitOps controller
    • Depends on: Flux Operator

Example output:

2025-11-04T15:07:00Z INFO Running stage... stage=apps

Usage

Full Bootstrap

just bootstrap on its own only lists the module's recipes (set default-list in bootstrap/mod.just). It no longer arms a bootstrap by itself. Run the full end-to-end flow with the cluster recipe:

just bootstrap cluster

This prompts for confirmation first (Bootstrap cluster? [y|N]) before running any stage.

This is equivalent to:

just bootstrap talos
just bootstrap kube
just bootstrap kubeconfig node
just bootstrap wait
just bootstrap namespaces
just bootstrap resources
just bootstrap crds
just bootstrap apps
just bootstrap kubeconfig

The first kubeconfig node call points kubectl directly at a control-plane node IP. Cilium's LoadBalancer doesn't exist yet at this point in the bootstrap, so there's no LB IP to route through. The final kubeconfig call (default cilium) re-fetches it pointed at the Cilium LB once Cilium is up and running.

Partial Bootstrap

Run individual stages as needed:

# Only deploy applications (assumes cluster is already bootstrapped)
just bootstrap apps

# Re-fetch kubeconfig
just bootstrap kubeconfig

# Re-apply namespaces
just bootstrap namespaces

List Available Commands

just --list

Dependency Management

Applications are deployed with explicit dependency chains using Helmfile's needs directive:

graph TD
    A[Cilium] --> B[CoreDNS]
    B --> C[Spegel]
    C --> D[Cert-Manager]
    D --> E[External Secrets]
    E --> F[OnePassword Connect]
    F --> G[Flux Operator]
    G --> H[Flux Instance]

Benefits:

  • Ensures correct deployment order
  • Prevents race conditions
  • Automatic retry on failures
  • Clear dependency visualization

Post-Sync Hooks

Certain applications use post-sync hooks to ensure dependent resources are ready:

Cilium Hooks

Cilium has two postsync hooks in bootstrap/helmfile.d/01-apps.yaml:

  1. Waits for Cilium CRDs to be available:

    kubectl wait --for=create \
  crd/ciliumbgpadvertisements.cilium.io \
  crd/ciliumbgpclusterconfigs.cilium.io \
  crd/ciliumbgppeerconfigs.cilium.io \
  crd/ciliumloadbalancerippools.cilium.io \
  --timeout=2m

  2. Server-side applies Cilium's own networking resources, so they exist before Flux takes over management of them:

    kubectl apply --namespace=kube-system --server-side \
  --field-manager=kustomize-controller \
  --filename=../../kubernetes/apps/kube-system/cilium/app/networks.yaml

OnePassword Connect Hooks

OnePassword Connect also has two postsync hooks:

  1. Waits for the ClusterSecretStore CRD to be available:

    kubectl wait --for=create \
  crd/clustersecretstores.external-secrets.io \
  --timeout=2m

  2. Server-side applies the ClusterSecretStore resource itself, so ExternalSecrets can start resolving onepassword-connect refs immediately:

    kubectl apply --server-side \
  --field-manager=kustomize-controller \
  --filename=../../kubernetes/apps/external-secrets/onepassword-connect/app/clustersecretstore.yaml

Values Template (DRY Principle)

The bootstrap uses a Go template to source Helm values from HelmRelease files:

File: bootstrap/helmfile.d/templates/values.yaml.gotmpl

{{ (fromYaml (readFile (printf "../../../kubernetes/apps/%s/%s/app/helmrelease.yaml" .Release.Namespace .Release.Name))).spec.values | toYaml }}

How it works:

  1. Constructs path to HelmRelease file
  2. Reads the YAML file
  3. Extracts .spec.values
  4. Returns as YAML

Benefits:

  • Single source of truth for Helm values
  • No duplication between HelmRelease and Helmfile
  • Easier maintenance and updates

Troubleshooting

Check Bootstrap Status

# Check if Talos nodes are ready
talosctl get members

# Check Kubernetes nodes
kubectl get nodes

# Check bootstrap pods
kubectl get pods -n kube-system
kubectl get pods -n cert-manager
kubectl get pods -n external-secrets
kubectl get pods -n flux-system

# Check Flux sync status
flux get kustomizations

Common Issues

1. "Failed to fetch kubeconfig"

Cause: Control plane not ready

Solution:

# Wait for control plane
sleep 30
just bootstrap kubeconfig

2. "Failed to apply namespace"

Cause: Cluster not accessible

Solution:

# Verify kubeconfig
kubectl cluster-info
export KUBECONFIG=$(pwd)/kubeconfig

3. "Failed to sync helmfile"

Cause: Missing CRDs or dependency failures

Solution:

# Check Helm releases
helmfile -f bootstrap/helmfile.d/01-apps.yaml list

# View detailed errors
helmfile -f bootstrap/helmfile.d/01-apps.yaml sync --debug

4. "vals eval failed" / secret resolution failed

Cause: 1Password CLI not authenticated (vals resolves ref+op:// refs via op)

Solution:

# Authenticate with 1Password
eval $(op signin)

# Verify authentication
op whoami

Debug Mode

Run just with debug output:

just --verbose bootstrap

Run helmfile with debug output:

helmfile -f bootstrap/helmfile.d/01-apps.yaml sync --debug

Advanced Usage

Customizing the Bootstrap

Adding New CRDs

Edit bootstrap/helmfile.d/00-crds.yaml:

releases:
    - name: my-operator
      namespace: operators
      chart: oci://registry.example.com/my-operator
      version: 1.0.0

Adding New Bootstrap Apps

Edit bootstrap/helmfile.d/01-apps.yaml:

releases:
    - name: my-app
      namespace: my-namespace
      chart: oci://registry.example.com/my-app
      version: 1.0.0
      values:
          - ./templates/values.yaml.gotmpl
      needs:
          - flux-system/flux-instance # Depends on Flux

Modifying Bootstrap Resources

Add secrets or resources under bootstrap/kustomize/apps/<namespace>/ and reference them from the namespace's kustomization.yaml. Use ref+op:// notation for 1Password-backed values:

---
apiVersion: v1
kind: Secret
metadata:
    name: my-secret
    namespace: my-namespace
stringData:
    value: ref+op://vault/item/field

Migration from Bash Scripts

The old bash-based bootstrap system (scripts/bootstrap-apps.sh) has been removed. If you're migrating from the old system:

Old Command

./scripts/bootstrap-apps.sh

New Command

just bootstrap cluster

Key Differences

Feature Old (Bash) New (Just)
Modularity Monolithic script Granular stages
Dependencies Manual ordering Helmfile needs
Logging Basic echo Beautiful gum output
Debugging Hard to debug Run individual stages
Values Duplicated DRY via templates
Maintenance ~300 lines of Bash Clean task definitions

References

Changelog

2025-11-04 - Bootstrap Modernization

  • Migrated from Bash scripts to just task runner
  • Implemented modular helmfile structure
  • Added DRY values templating
  • Added explicit dependency management
  • Improved logging with gum
  • Removed 325 lines of Bash code
  • Added comprehensive documentation