Troubleshooting¶
What broke while building and running this pipeline, grouped by area. Each entry is a symptom, its cause, and the fix.
For the full data flow and design rationale, see architecture.md.
Ingestion path¶
- First Function run returns
403from the Logs Ingestion API. - Cause: the Function's managed identity was just granted Monitoring Metrics Publisher on the Data Collection Rule (DCR), and that role assignment can take ~30 minutes to propagate.
-
Fix: expected on a cold deploy, not a defect. Events buffer on the queue until the Function can write, and
deploy.shretries the code publish. Wait out the propagation window. -
Uploads fail with HTTP
404. - Cause: a wrong DCR immutable ID, or a stream-name mismatch. The stream must be
Custom-OpenZitiEvents_CLin the DCRstream_declaration, the DCRoutput_stream, and the Function's upload call. -
Fix: make all three identical, and confirm the Function is reading the DCR immutable ID (not the DCR resource ID) from its app settings.
-
Events go missing under bursty load.
- Cause: the OpenZiti
servicebussink does a non-blocking send and drops events when itsbufferSizefills. Watch for theservice bus queue fulllog line on the controller. -
Fix: raise
bufferSizein the sink config for bursty runs. -
Controller reconnects endlessly and no messages arrive.
- Cause: a bad SAS key or wrong queue name. The sink treats
401/403/404as connection errors and enters an infinite reconnect loop rather than failing fast, so the symptom is reconnect churn with no throughput. -
Fix: validate the Send connection string and the queue name first; they are the usual culprits.
-
The
servicebushandler config fields are undocumented. - Cause: the OpenZiti config reference does not yet document the
servicebushandler. -
Fix: the field names (
connectionString,queue,bufferSize,format) are verified from the v2.0.0 source (controller/events/servicebus_logger.go) and the packaged example config. -
The DCR transform rejects
coalesce(). - Cause: the ingestion-time
transformKqlruns a restricted KQL subset that has nocoalesce()and rejects it withRuntime scalar function provider not found for function: coalesce. -
Fix: use
iif(isnotempty(tostring(x)), tostring(x), tostring(y)). This is needed to reconcile OpenZiti'sevent_type(snake_case) vseventType(camelCase, onentityChange) split. -
The DCR create races the custom table.
- Cause: the DCR validates its
outputStreamagainst the_CLtable at create time, and the analytics rules also need the table to exist. - Fix: keep the DCR's
depends_onon the table, and expect a short queryable-lag after the table is created - the deploy retry loop covers it.
Targeting an existing workspace¶
- Custom table create fails with
400 InvalidParameteron retention. - Cause: the table's
totalRetentionInDaysmust be >= the target workspace's retention. Pointing at a workspace with, say, 90-day retention while the table is set to 30 fails. A self-created workspace never hits this because it is created at the same retention. -
Fix: set
retention_in_daysto match or exceed the target workspace. Note thataz ... table create --retention-timesets only interactive retention and can mask the problem; theazapibody sets total retention explicitly. -
Analytics rules fail:
One of the tables does not exist. - Cause: the rules reference
OpenZitiEvents_CLonly inside a KQL string, so Terraform schedules them in parallel with the table, and Sentinel validates the query at create time before the table is ready. -
Fix: the rules
depends_onthe table (in addition to onboarding). Keep that dependency. -
DCR role assignment fails with Contributor.
- Cause: creating the role assignment needs
Microsoft.Authorization/roleAssignments/write, which Contributor excludes. -
Fix: grant User Access Administrator or Owner on the scope alongside Contributor. Everything else (table, DCR, Function, controller) is covered by Contributor.
-
Pre-flight checklist. Not a failure entry - four checks worth running before the first deploy into a governed subscription:
- Resolve the workspace ARM ID from its customerId GUID:
az monitor log-analytics workspace list --query "[?customerId=='<guid>']". - Confirm resource-provider registration.
- Scan
az policy assignment listfor deny-public-endpoint / force-private policies that would block public Data Collection Endpoint (DCE) ingestion or the demo controller. - Smoke-test container capacity live - it is not a queryable quota. Create a
throwaway 1 vCPU / 2 GB container with
az container create, confirm it reachesRunning, then delete it.
Function hosting¶
- Flex Consumption deploys and publishes but never runs.
- Cause: on Flex Consumption the host published cleanly yet never executed the worker - zero Application Insights telemetry, and the queue never drained.
-
Fix: use a Y1 Consumption plan, which consumes the queue immediately.
-
Service Bus trigger never fires with a managed-identity connection.
- Cause: the Consumption scale controller cannot peek the queue over managed identity to scale from zero, so messages sit unconsumed.
-
Fix: use a Listen SAS connection string for the trigger. The Logs Ingestion upload still uses managed identity.
-
The Function host fails to initialise.
- Cause:
AzureWebJobsStorageis a keyed connection string; a keyless storage account leaves the host unable to start. - Fix: set
shared_access_key_enabled = trueon the Functions runtime storage account.
Azure Container Instances (ACI) demo controller¶
- PKI bootstrap fails on the Azure Files mount.
- Cause: ziti's PKI store hard-links the intermediate CA bundle (
os.Link), which CIFS does not support, and bbolt/raft uses mmap, which is unsafe over CIFS. Bootstrapping directly on the share wedges half-way. -
Fix: bootstrap on container-local disk and snapshot only the identity (PKI + config) to the share; restore the snapshot on restart. The raft database is rebuilt on each start (
clusterInitis idempotent). -
The identity snapshot itself wedges bootstrap.
- Cause:
cp -aimplies--preserve=linksand tries to recreate the PKI hard links on the share, dying part-way - afterconfig.ymlis already copied. -
Fix: snapshot with
cp -rL(nevercp -a) and copyconfig.ymllast, as the restore-branch sentinel. Any partial state on the share wedges bootstrap permanently; if in doubt, wipe the share and let it re-bootstrap. -
The container dies immediately with empty logs.
- Cause: the image's
bootstrap()writes DEBUG lines to fd 3, which the stock entrypoint opens but a barebash -cscript does not; underset -ethe first DEBUG line kills the container. -
Fix: the wrapper script must
exec 3>&1. -
makePkihard-errors on unset cluster variables. - Cause: the image ENV does not set the cluster vars;
makePkidereferencesZITI_BOOTSTRAP_CLUSTERunder nounset and hard-fails on an empty trust domain. -
Fix: set
ZITI_BOOTSTRAP_CLUSTER,ZITI_CLUSTER_TRUST_DOMAIN, andZITI_CLUSTER_NODE_NAMEexplicitly in the container environment. -
az container logsreturns mostlyNonefor a crash-looping container. - Cause: the log window between restarts is only seconds.
-
Fix: poll tightly and keep the longest snapshot; read
instanceView.eventsandpreviousState.exitCodefor the exit history. -
Periodic
handshake failednoise from10.92.0.xon port 1280. - Cause: ACI infrastructure probes open TCP but never complete TLS.
-
Fix: none needed - this is expected noise, not a defect.
-
The controller keeps billing when idle.
- Cause: there is no auto-shutdown for ACI (the VM shutdown schedule is VM-only), so the group runs continuously.
- Fix:
az container stopwhen idle; the identity survives via the Azure Files snapshot, so it comes back up without re-minting its PKI.
Lifecycle¶
- Redeploying a Sentinel rule fails with
409"recently deleted". - Cause:
azurerm_sentinel_alert_rule_scheduleduses the rule name as its ID, and a deleted ID stays reserved through a cooldown longer than a few minutes. -
Fix: this repository suffixes rule names with the per-deploy random suffix, so redeploys always use fresh IDs and never collide.
-
Same-subscription redeploy of the Function fails with
401quota. - Cause: a just-deleted Y1 Consumption plan's quota is not released immediately; the limit briefly reads 0 (reported anywhere from ~30 min to hours). This is a subscription capacity limit, not code and not policy. A different subscription is unaffected.
-
Fix: wait for the quota to recover, or redeploy into a fresh subscription. The deploy retry loop deliberately does not wait this out (it would block for an unbounded time).
-
Budget create fails on
start_date. - Cause: a monthly budget's
start_datemust be the first of a month and not far in the past. -
Fix: it is derived to the first of the current month via
formatdate(timestamp()); do not hardcode a past date. -
A soft-deleted workspace lingers after teardown.
- Cause: Log Analytics workspaces soft-delete for 14 days. The provider sets
permanently_delete_on_destroy, but that only runs on the cleanterraform destroypath. -
Fix: teardown also purges any stray soft-deleted workspace of the same name. A leftover from a failed destroy is harmless and auto-purges in 14 days.
-
terraform destroyfails on the storage share, orphaning the workspace. - Cause:
azurerm_storage_sharedestroy goes through the storage data plane (token audiencestorage.azure.com), which can fail even when ARM auth is fine (does not exist in MSAL token cache), failing the whole destroy. The fallbackaz group deletethen leaves the workspace soft-deleted. -
Fix: the provider sets
storage_use_azuread = true. If the fallback path still orphans a soft-deleted workspace, it is harmless and auto-purges in 14 days. -
Teardown reports success while resources remain.
- Cause: a failed
terraform destroystill strips root outputs from state, so on a re-runterraform output -raw resource_group_nameprints its "No outputs found" warning on stdout instead of a name. A completion check against that text, or against the wrong default subscription, or one that treats anazerror as "gone", reads as a pass. -
Fix: teardown resolves the subscription, resource group, and workspace names from configuration and
terraform.tfvarsthroughterraform console, refuses to run if they do not look like a subscription GUID and anrg-name, passes--subscriptionon everyazcall, and reports success only whenaz group existsreturns an explicitfalse. -
az monitor log-analytics query -wreturns nothing / errors on the ID. - Cause: the query API wants the workspace customerId GUID, not the ARM resource ID.
- Fix: resolve it first
(
az monitor log-analytics workspace show --ids <arm-id> --query customerId);verify.shalready does this.
Event semantics¶
- Authentication
Successdisagrees withevent_type. - Cause: in OpenZiti v2.0.0, an authentication event can carry
event_type: "success"withsuccess: false- observed consistently, with every event in a run reportingsuccess: falseregardless ofevent_type. - Fix: key detections on
EventType, notSuccess. The shipped auth-failure rule keys onEventType in ("fail", "failed").
Subscriptions and providers¶
- Small VM SKUs are unavailable (
NotAvailableForSubscription). - Cause: some subscription offers restrict small VM SKUs entirely (restriction type Location) even with healthy vCPU quota. This is a platform SKU-enablement matter tied to the offer, not an Azure Policy.
-
Fix: resolve it with a SKU/quota support request or a VM-capable subscription, or run the demo controller on ACI, which is why this repository does.
-
terraform planworks but apply or teardown fails with401 RequestDisallowedByAzure. - Cause: the tenant enforces multifactor authentication for Azure write
operations. A plain
az logintoken is enough to read, soplansucceeds, but writes are rejected withRequestDisallowedByAzure ... without authenticating through MFA. -
Fix: log in again with a claims challenge that demands MFA:
-
First apply fails with
MissingSubscriptionRegistration. - Cause: with
resource_provider_registrations = "none"on a governed subscription, the required providers are not registered. - Fix: pre-register them, sequentially (a parallel loop can silently
no-op):
Microsoft.Network,Microsoft.Storage,Microsoft.ServiceBus,Microsoft.OperationalInsights,Microsoft.Insights,Microsoft.SecurityInsights,Microsoft.OperationsManagement,Microsoft.Web,Microsoft.ContainerInstance, andMicrosoft.Consumption.Microsoft.Consumptionis absent from everyazurermregistration preset and is needed only for the optional budget alert.