How it works¶

The path a record takes¶

sequenceDiagram
    participant App
    participant AI as Application Insights
    participant LA as Log Analytics
    participant DE as Data Export
    participant Blob as WORM container

    App->>AI: TrackEvent("Alteration", {...})
    Note over App,AI: buffered, or flushed immediately<br/>when severity is Critical
    AI->>LA: ingest into AppEvents
    Note over LA: queryable within 2-5 minutes
    LA->>DE: continuous export
    Note over DE: ~30 min to provision,<br/>then batched every 5 minutes
    DE->>Blob: append to PT5M.json
    Note over Blob: retention policy already attached<br/>before the first write

Why custom events¶

The application emits audit records as Application Insights custom events, which land in the AppEvents table in Log Analytics.

That choice is doing real work. AppEvents is a standard table, and Data Export only supports standard tables. A custom table created through the Logs Ingestion API would need a data collection endpoint, a data collection rule, a schema to maintain and a managed identity with the right role - and it would still export the same way at the end of it. Custom events get to the same place with a single SDK call and no ingestion infrastructure of your own.

The practical consequence is that no application change is needed beyond the TrackEvent calls you would write anyway. If your application already emits custom events, the entire pipeline is infrastructure.

az monitor app-insights component show --ids "$APP_INSIGHTS_ID" \
  --query properties.WorkspaceResourceId -o tsv

The two retention clocks¶

These are independent, and conflating them is the most common design mistake.

The workspace is the hot window: where you investigate, hunt and build detections, and where a long retention gets expensive fast. The blobs are the long-term record: cheap, immutable, and awkward to query.

Set the workspace window to however long you realistically investigate over - 30 to 90 days for most people - and let the blobs carry the statutory tail. There is no benefit to paying workspace rates for six years of data you will query twice.

What Data Export actually does¶

Data Export is a workspace-level rule that continuously copies rows from named tables into a storage account as they arrive. It is not a scheduled job and there is nothing to run.

Provisioning takes around 30 minutes. Data generated before the rule is live may never appear in storage. This is the single most common "it is broken" report, and it is not broken. Wait the 30 minutes before investigating anything.

After that, latency is around five minutes. Rows are batched into five-minute windows.

One container per table, named am- followed by the lowercased table name. AppEvents becomes am-appevents. This is not configurable. The templates here build the container list from the same array that feeds the export rule, so the two cannot drift. If they ever do, export creates its own container with no retention policy on it, and nothing tells you. Alert on it.

Blob layout¶

WorkspaceResourceId=/subscriptions/<sub>/resourcegroups/<rg>/providers/microsoft.operationalinsights/workspaces/<ws>/
  y=2026/m=07/d=30/h=16/m=25/PT5M.json

Three things about that path catch people out:

1. m= appears twice and means different things. The segment after y= is the month. The segment after h= is the minute. A parser that takes the first m= as the month is right two thirds of the time and wrong the rest, with nothing to show for it.
2. The workspace resource ID in the path is lowercased, including the resourcegroups and microsoft.operationalinsights segments. A prefix match built from the resource ID as the portal displays it will not match.
3. Overflow files sit alongside. Where a five-minute window exceeds 50,000 appends, the surplus lands as a numbered sibling in the same folder. Enumerate the folder; do not assume one file per window.

The contents are newline-delimited JSON with no enclosing array - one record per line. jq -s or a streaming reader, not a plain json.load.

Why containers are created before the export rule¶

The templates create every container, and attach its immutability policy, before the export rule exists.

The alternative - deploy the rule, let export create containers on demand, apply policies afterwards - has a window of roughly thirty minutes to several hours in which records are landing in containers that carry no retention policy. Applying a policy afterwards protects the container from that point forward. It does not retrospectively protect what is already in it.

For an archive whose whole value is that nothing in it can have been altered, "everything except the first few hours" is a meaningful gap, and an awkward one to explain later.

Immutability mechanics¶

allowProtectedAppendWrites must be true. Exported blobs are append blobs, extended over the life of their five-minute window. With protected appends disabled, the first append after the policy takes effect is rejected and export stops. This is not a weakening of immutability: appends can add new blocks but cannot modify or delete existing ones.

The retention clock runs from last modification, not creation. Because these are append blobs, a blob becomes eligible for deletion the retention period after the final append to it, not after it was created. A six-year policy on a blob appended to for five minutes expires six years and five minutes after it first appeared.

Unlocked and locked differ in one specific way. Both enforce retention against normal deletes. Only locked survives an administrator who wants the data gone, and only locked cannot be undone by you either. Deploy unlocked, confirm the pipeline works, then lock.

Identity, and one trap¶

Easy Auth terminates sign-in at the App Service edge and forwards the signed-in principal as X-MS-CLIENT-PRINCIPAL-NAME. The app promotes that header to context.User, and the telemetry service writes it to the event's property bag as UserId.

Why sampling is disabled¶

Adaptive sampling is on by default in the Application Insights ASP.NET Core SDK. It discards a proportion of telemetry under load and scales the survivors' ItemCount so aggregate metrics stay roughly right.

That is a good trade for performance monitoring and a bad one here. A sampled audit archive is not a complete account of what the application did, and nothing at query time can recover a record that was never sent. Worse, it fails quietly: the archive looks complete, and its incompleteness only surfaces when someone asks for a specific event that was dropped.

The demo app therefore constructs the SDK with EnableAdaptiveSampling = false. If you take one thing from this repository into your own application, take that.

A related habit: aggregate with sum(ItemCount) rather than count(). With sampling off the two agree. If they ever disagree, sampling has been turned back on somewhere.

Network posture¶

The retention tier is deployed with its public endpoint switched off. A blob private endpoint, registered in a private DNS zone, is the only route a client has to it.

The thing to understand before changing any of this is that the private endpoint is not on the path the records travel. Data Export writes through the Azure Monitor platform, not as a network client of the storage account. So the endpoint governs who can read the archive; it has no bearing on whether the archive fills.

flowchart LR
    A[Application] --> B[Application Insights]
    B --> C[Log Analytics]
    C -->|"Data Export<br/><small>trusted-services path</small>"| D[(WORM storage<br/>public access disabled)]
    E["Operator or<br/>retrieval tool"] -->|"private endpoint<br/><small>10.x.x.x</small>"| D
    F["Anything on<br/>the internet"] -.->|blocked| D

What it costs you¶

You cannot read the archive from your own machine any more, which is the point, but it does mean the ordinary verification path stops working. scripts/verify.sh reports the containers as unreachable and correctly says so rather than blaming permissions.

Use scripts/verify-private.sh instead. It creates a user-assigned identity, grants it Storage Blob Data Reader, runs a throwaway container in the verifier subnet, prints what that container could see, and deletes itself.

Two subnets, not one¶

The private endpoint subnet needs privateEndpointNetworkPolicies disabled or the endpoint deploys and then receives nothing. The verifier subnet is delegated to Azure Container Instances, and a delegated subnet cannot host anything else. They therefore cannot be the same subnet.

main.bicep creates both, because it is the self-contained demo. retention-only.bicep takes an existing subnet and DNS zone, because a landing zone owns its address space and a template that invents one produces an island nothing can route to.

TLS¶

The storage account is fixed at TLS 1.2. TLS1_3 appears in the ARM enum and in the Azure Verified Modules allow-list, but the resource provider rejects it:

FeatureNotSupported: Feature MinimumTlsVersion 1.3 is not supported.

App Service is the opposite case: 1.3 is accepted and applied. The app defaults to 1.2 anyway, because requiring 1.3 refuses clients that cannot negotiate it, which is a decision about your callers rather than about hardening.

Limits¶

• A workspace supports at most 10 active export rules.
• A storage account can be the destination of only one rule on a given workspace.
• Premium storage accounts are not supported as export destinations.
• The storage account must be in the same region as the workspace. Export to another region is rejected outright.
• Every table named in a rule must already exist. A table that has never received data does not exist, and naming it fails the whole rule rather than just that table.