A. B. Khan

Turning telemetry into signed evidence

The Mesh Series · Article 27

Compliance evidenceSupply-chain signingLGTM stackDrift & postureMulti-cluster & residencyAudit loggingDistributed tracingPCI DSS

Cover: Turning telemetry into signed evidence

Keeping the record and proving it’s the record are two different jobs. A signature does the second one.

You have the logs. That was Article 24. You have the stores that answer four different questions — Loki for the record, Tempo for the causal chain, Mimir for the posture over time. That was Article 25. You even have identity threaded through every trace, so a request can be tied back to the human who made it. That was Article 26.

And none of it is evidence yet.

Here is the sentence that ends the demo and starts the audit: “Prove nobody changed this.” You point at a Grafana panel showing a clean access record. The assessor doesn’t dispute what’s on screen. They ask how they’d know the row was there yesterday, and wasn’t quietly rewritten this morning before you took the screenshot. You open Loki and run the query live. Same problem — the person who runs the store can edit the store. A log an operator can edit is not an audit trail. It’s a note that happens to be true today.

That gap — between keeping the record and proving the record is the one you kept — is the whole of this article.

Diagram: signed evidence

Tamper-evident is a different job from tamper-proof

Start by separating two words people use as if they mean the same thing.

Tamper-resistant means you tried to stop the change: WORM storage, S3 Object Lock in compliance mode, a retention lock nobody — not even the account root — can shorten. Prevention.

Tamper-evident means you don’t try to stop the change; you make any change impossible to hide. You sign the artifact. Alter a byte and the signature no longer verifies. Detection.

They solve different failure modes, and a regulated evidence story needs both. A signature tells you if someone changed the evidence. It does nothing to stop someone deleting it — an unsigned gap is just a gap, and “we have no logs for that window” fails the audit as surely as a forged line does. Object Lock stops the deletion but proves nothing about whether the retained bytes are the originals or a doctored replacement written before the lock engaged. One detects, the other prevents. Reach for either alone and an assessor finds the other half missing.

This article is about the detection half, because that’s the half almost nobody builds and the half that turns a store into evidence. The prevention half is a storage-tier checkbox; the detection half is a signing discipline.

Why an assessor stops trusting the dashboard

The regulator isn’t being difficult. They’ve seen doctored logs. So the burden isn’t “show me the data,” it’s “show me the data and prove it’s the data you had at the time.”

In India, CERT-In wants incident logs retained 180 days, within Indian jurisdiction, producible on demand. PCI-DSS 4.0 is blunter: Requirement 10 says track and monitor all access, and 10.5 says protect those audit trails so they cannot be altered — file-integrity monitoring or change detection on the logs themselves is written into the requirement. The audit record has to be defended as hard as the thing it records.

SAMA’s Cyber Security Framework raises the bar again, because it grades you on a maturity model. You aren’t proving a control fired once during the assessment window; you’re proving it has operated continuously, and that the record of it operating hasn’t been curated after the fact. A screenshot is the weakest artifact in that room — it’s a picture of a claim, produced by the party making the claim, at a time of their choosing. The immutable, signed store is the strong artifact. Everything in this series has been walking toward being able to hand an assessor the second thing instead of the first.

There’s a residency twist that makes this sharper in both markets. Your evidence has to be tamper-evident and it has to stay in-country. That rules out the reflexive answer of shipping everything to a SaaS observability vendor whose signing and storage live wherever their region happens to be. Self-hosted stores keep the bytes in jurisdiction; signing keeps them provable. You need the pair.

Where signing sits in the stack you already built

The good news is you’re not adding a new trust anchor. You’re spending the ones the series already established.

The collection path is proven: mesh mTLS carried the telemetry (Article 14), the access logs already name a SPIFFE identity and an Entra principal (Article 24), and the traces are threaded with a real subject (Article 26). What’s missing is the last step — turning a live queryable store into a frozen, signed artifact at a point in time.

The shape is a periodic evidence job. On a schedule — nightly, per control, whatever the framework’s window demands — a small workload runs a scoped, deterministic query against Loki, Tempo, or Mimir, serializes the result into a canonical bytes-for-bytes-reproducible document (JSON for machines, a rendered PDF for the human auditor), and signs that document. The signature is detached: the evidence and its proof travel as separate files, so the pack is human-readable and the proof is independently verifiable.

Two properties make the signature audit-grade rather than decorative.

First, who signed it. This is where the passwordless workload story (Articles 8 and 15) pays off a second time. The evidence job doesn’t hold a long-lived signing key in a config file — that key would be exactly the “secret you’re afraid to rotate” this series has been killing off. It authenticates with its federated workload identity and gets a short-lived signing certificate keyed to that identity. The signer is a named, revocable machine identity, not an anonymous key that whoever-holds-the-string can wield. The confused-deputy problem you spent Article 17 avoiding doesn’t reappear at the signing step.

Second, when it was signed. A bare signature over a short-lived certificate has an awkward property: the cert expires, and later you can’t tell whether the signature was made while the cert was valid or forged afterward. The fix is a trusted timestamp from an RFC 3161 timestamp authority — a second party that cryptographically vouches for the moment of signing. The signature proves who. The timestamp proves when. For an evidence pack whose entire value is “this is what the system looked like on the night of the fourteenth,” the when is not a nice-to-have. It’s half the claim.

Show the code

The signing primitive is Sigstore’s cosign. Most people meet it signing container images; the part that matters here is that it signs arbitrary blobs — a JSON evidence pack, a PDF, anything with a digest. As of cosign v3.1.1 (June 2026), the Sigstore bundle format is the default: signature, signing certificate, and transparency-log inclusion proof land in one file you can verify offline.

Code: listing

If the pack is one byte different from what was signed, verification exits non-zero. That’s the entire property. The dashboard could lie; this cannot, without the lie being visible. And note that verification pins which identity was allowed to sign, so a valid signature from the wrong workload still fails.

For evidence you want to carry meaning, not just integrity, wrap it as an in-toto attestation instead of a raw signature. The DSSE envelope binds a predicate — structured metadata — to the subject (the evidence pack’s digest). Your predicate is the compliance claim: which control, which framework, which window, which query produced it.

Code: listing

The C# side is the boring, load-bearing part: assemble the pack deterministically, because a signature is only meaningful over reproducible bytes. Same query, same window, same serialization — every time.

Code: listing

The ordering line is not incidental. If the row order wanders between runs, the digest wanders, and you’ve built a signature over noise. The evidence job’s one discipline is determinism.

Security · Audit · Integration

Security property gained. Tamper-evidence and non-repudiation over the audit record itself. Any post-hoc edit to a signed pack is detectable by anyone with the public trust root; the signer is a named, short-lived, revocable workload identity, not a shared key. Trusted timestamping binds the signature to a moment, so an expired signing certificate doesn’t weaken an old, legitimate signature.

Audit evidence it produces. A self-contained bundle per control per window: the frozen pack, its detached signature, the signing certificate, an RFC 3161 timestamp token, and — if you anchor it — a transparency-log inclusion proof. That set answers “is this the original, who produced it, and when” without the auditor trusting your infrastructure or your good intentions.

How it integrates. The signing identity is the same federated workload identity from Articles 8 and 15 — no new secret. The stores are the self-hosted, in-country LGTM stack from Article 25, fronted by WORM retention so the prevention and detection halves both hold. The transparency log is optional and external; the signature stands on its own offline.

Alternatives, and when you’d pick them

Sigstore (cosign + in-toto, optionally Rekor). The default recommendation, and the one with real ecosystem gravity — Sigstore-signed in-toto attestations are what Homebrew, PyPI, Maven Central, and others already ship. Keyless signing via short-lived certificates fits the passwordless posture exactly. Rekor v2 went GA in October 2025 as a tile-backed log on the Tessera framework — cheaper to run than v1, which matters if you self-host the transparency log for residency. Pick this unless you have a specific reason not to.

A hash-chained append-only log (Merkle tree you run yourself). Each entry commits to the previous, so the sequence is tamper-evident without an external signer. Legitimate, and sometimes the right answer when you can’t take an external dependency. But now you operate the transparency infrastructure and defend its root — you’ve rebuilt a chunk of Rekor. Pick it only when a hard air-gap or a no-external-trust mandate forces your hand.

Cloud-native log integrity — AWS CloudTrail digest files, Azure immutable blob storage. Real, and if you’re already all-in on one cloud’s audit primitives, use them. The catch is scope: they prove integrity of that provider’s control-plane logs, not your mesh access logs or your Entra-annotated traces. And the residency question follows you — immutable storage in the wrong region is still the wrong region.

Long-lived KMS signing key instead of keyless. Simpler mental model, one key in an HSM. But it reintroduces exactly the artifact this series has spent nine articles removing: a durable secret with a rotation problem and a “who can wield it” problem. Sometimes an HSM-backed key is a genuine regulatory expectation — then do it, with the key in the HSM and never in config. Otherwise short-lived certificates are strictly less to guard.

“We put it on a blockchain.” Almost always the wrong tool wearing the right word. If what you need is “prove this bytes-blob existed at this time and hasn’t changed,” a signature plus a trusted timestamp plus a transparency log already does it, cheaper and with tooling an assessor recognizes. Reach for a distributed ledger only when the actual requirement is multi-party consensus with no trusted timestamp authority available — which, inside one institution’s compliance program, it isn’t.

Have you ever had an auditor (RBI inspection, SAMA assessor, PCI QSA) actually reject a screenshot or an editable export as evidence — the moment “we have the logs” wasn’t enough? What did they ask for instead, and what did you end up producing?

From the toy to the standard

The one-liner above signs a single file. The industry-grade version is a system, and it’s a small one.

A scheduled evidence job — a Kubernetes CronJob is enough — runs per control, queries the scoped window, freezes the deterministic pack, signs it with its federated identity plus a trusted timestamp, wraps it in an in-toto predicate that names the framework and control, and writes the whole bundle to a store under Object Lock with a retention period matched to the framework: CERT-In’s 180 days, PCI’s one year with ninety days readily available, whatever SAMA’s window works out to for the control. Prevention from the storage tier, detection from the signature, provenance from the predicate, residency from where the bucket lives. Verification is a script you hand the auditor, not a person you hand them.

That’s the demo-to-audit gap closed for one control. Do it for the set of controls the framework enumerates and you have something the earlier articles couldn’t produce on their own: not “here are our logs,” but “here is a signed, timestamped, in-country attestation that this control operated over this window, and here is the command to prove I didn’t touch it.”

One honest caveat, because it tees the next article. A signed pack proves the record is intact. It doesn’t prove the record is complete — that the control was actually running the whole window and didn’t silently stop holding at 3 a.m. while the store faithfully recorded nothing wrong. Tamper-evidence guarantees you didn’t edit the evidence; it says nothing about whether the thing being evidenced kept working. That’s drift, and detecting a control that quietly stopped being true is Article 28.

A screenshot is a claim. A signed pack is a proof.

Next — Article 28: Continuous posture and drift. Proving a control didn’t just pass once, but kept holding — and catching the moment it silently stopped.

Key skills & topics in this article

Compliance evidenceSupply-chain signingLGTM stackDrift & postureMulti-cluster & residencyAudit loggingDistributed tracingPCI DSS

  • #ServiceMesh
  • #Istio
  • #Azure
  • #dotnet
  • #ZeroTrust
  • #Kubernetes

Also on LinkedIn.