Disaster recovery for a regulated mesh — recovering the proof, not just the app
The Mesh Series · Article 40 · Part G
Compliance evidenceIstioDisaster recoveryGitOpsWorkload identitymTLSSupply-chain signingAuthorizationPolicy

A regional outage takes your primary cluster offline at 03:10. By 03:40 the platform team has a cluster in the paired region, Argo CD has reconciled the application manifests, and the dashboards are green. At 09:00 the ICT risk function asks for the evidence that the payment service was still enforcing STRICT mTLS and the default‑deny policy during the forty minutes in between — and for proof that the evidence store you restored from is the evidence store you had.
Green dashboards. No proof. That is not a recovered platform. That is a platform that works and cannot say so.
One idea: DR isn't your app; it's your ability to still prove things afterwards
Every DR plan recovers the application tier. Most recover the data tier. Almost none recover the two planes a regulated mesh actually depends on: the trust plane — the root of trust every workload identity chains to — and the evidence plane — the signed, timestamped, write‑once records that prove the controls held. Those two are what DORA's Article 11 is really asking for when it says response and recovery, and what Article 12 means when it insists that restoration be tested, on segregated systems, not just that backups exist.
The trap is specific and it bites in the first ten minutes. If you reinstall istiod into a fresh cluster the default way, it mints a new self‑signed root and starts signing workload certificates from it. Every restored workload now carries an identity that chains to a root nothing else trusts — not the other cluster, not the restored copies of your own policies' expectations, not the evidence you wrote yesterday. STRICT mTLS still says STRICT. It is simply enforcing a trust domain that did not exist at 03:09. That is a mesh‑wide control‑failure event with its own reporting clock (Article 38), and it looks exactly like success on a dashboard.
The fix is not clever. It is order. Recover the root of trust first, and nothing after it has to be re‑trusted.

Six steps, in order. The order is the plan.
Why it matters: a restored control you cannot prove is a control you do not have
DORA Article 12(2) wants restoration tested, periodically. Article 12(3) wants it tested on systems physically and logically segregated from the source. Article 12(6) wants redundant ICT capacity for anything that is not a microenterprise. NIS2 Article 21(2)(c) asks for business continuity, backup management and disaster recovery as a maintained capability. None of those clauses mention a service mesh. All of them bite on it, because the mesh is where your most defensible controls live and the mesh is what a naïve DR runbook silently replaces.
Three things go wrong, in the order a regulator discovers them.
The root key was never in the backup. It was in istio-system as a secret nobody labelled, or it was in a Key Vault in the region that just went dark, or — worst — it was generated by istioctl install on day one and nobody has seen it since. Article 39 put the root in its own jurisdiction. This article makes sure it is offline in that jurisdiction: generated on a machine that is not the cluster, stored in an HSM or a sealed vault, used once to sign the intermediate that istiod actually runs with. A DR test that cannot re‑issue that intermediate from the offline root is a DR test that has already failed.
The policy came back after the workloads. Argo CD reconciles what it is pointed at, in the order it finds convenient. If Deployments land before the AuthorizationPolicy and PeerAuthentication objects, you have a window in which restored workloads run with no mesh policy at all — mTLS permissive, authorisation allow‑all. The window may be seconds. The window is in the log. Article 16 said deny‑all and watch what screams; in DR that becomes deny‑all first, then let the workloads scream in a cluster that is already closed.
The evidence you restored from is not the evidence you had. This is the one almost everyone misses. Restoring a Loki bucket or an evidence pack from backup is a write. A write is a moment at which the record could change. If the Article 27 signatures do not verify after restore — the hash does not match, the timestamp authority's token is missing, the Rekor inclusion proof does not resolve — you do not have a restored evidence ledger. You have a tampering event, discovered by you, during your own DR test. That is reportable. It is also, if you planned for it, the most convincing piece of evidence in the whole exercise: we verified our own ledger and here is the verification.
And one trap that Article 28 planted: self‑healing is an evidence shredder. The moment Argo CD's auto‑sync reaches a half‑recovered cluster it starts "correcting" drift — which includes overwriting whatever state the outage left behind, before anyone captured it. Record before you correct. Pause auto‑sync, snapshot the live state, sign it, then let GitOps reconcile.
How it comes together: six steps, each one a precondition of the next
1 — Root of trust. The offline root re‑issues an intermediate for the recovery cluster, or the previously issued intermediate is restored from a backup that was itself encrypted to a key held in the recovery region. The four files — ca-cert.pem, ca-key.pem, root-cert.pem, cert-chain.pem — go into the cacerts secret in istio-system before anything else is installed. The SPIFFE trust domain is now continuous across the outage: same root, same trustDomain, same identities.
2 — Control plane. istiod installs and, finding cacerts, signs from the restored intermediate rather than minting its own root. Verify this before proceeding: istioctl proxy-config secret on the first restored pod should show a chain ending in the root you expect. Do not move to step 3 on a dashboard.
3 — Identity plane. The cluster's OIDC issuer URL is part of its Entra workload‑identity federation (Article 15). A new cluster has a new issuer. Either the federated credentials in Entra are updated to the recovery cluster's issuer — a change that must itself be logged and signed — or the recovery cluster was pre‑registered as a second federated credential months ago. The second is the standard; the first is the Tuesday‑morning scramble.
4 — Policy from Git, deny first. Argo CD applies PeerAuthentication (STRICT) and the namespace‑wide empty‑spec AuthorizationPolicy from Article 16 as a sync wave before any workload. Fail closed. Then the named allow rules. Then, and only then, the Deployments.
5 — Workloads. Pods start, request SVIDs, get them from the restored intermediate, and the same AuthorizationPolicy rules that held yesterday hold today, against the same principals. Nothing had to be re‑trusted because nothing changed.
6 — Evidence ledger. Loki, Tempo and the evidence bucket fail over to their in‑region replicas — replicas that carry the same Object Lock retention, because an immutable primary with a mutable replica is a mutable store with extra steps. Then verify: every evidence pack's cosign signature, timestamp token and Rekor inclusion proof is checked against the restored bytes. A verification report, itself signed, closes the DR test.
The whole sequence is Article 12(2)'s "periodic testing of restoration," done in a way that produces evidence of the test rather than a ticket that says it happened.
Show the config: back up the trust plane, verify the evidence plane
First, make the trust plane and the mesh policy part of the backup, explicitly. Velero backs up what you tell it; by default it does not know that one secret in istio-system is the thing every identity chains to.

Then, after failover, prove the ledger is the ledger. This runs against the restored bucket and refuses to finish green unless every pack verifies.

Security · Audit · Integration
Security: the root never enters a cluster and never enters a backup; the recovery cluster signs from an intermediate the root re‑issued, so the trust domain survives the outage without anyone copying a root key across a region boundary.
Audit: the DR test ends with a signed verification report over every evidence pack — Article 12(2)'s restoration test, proved rather than ticketed — and a failed verification is handled as an incident, not retried until it passes.
Integration: Velero carries the intermediate and the policy objects; Argo CD sync waves apply deny‑first; Entra holds a pre‑registered federated credential for the recovery cluster's issuer; the evidence stack fails over to an Object‑Locked replica rather than being restored from a copy.
Alternatives, and when you'd pick them
Active‑active multi‑cluster, no failover. Article 39's two‑cluster shape with both sides live means there is nothing to "recover" — traffic moves, identities already exist on both sides. Pick it when the two regions are allowed to share a trust domain (same jurisdiction, same regulator) and you can afford two full control planes. Across a residency line it reintroduces exactly the shared root Article 39 argued against.
Let the managed mesh do it. The AKS Istio add‑on and OpenShift Service Mesh manage the control plane and, to a degree, its CA. That removes step 2 from your runbook and adds a line to your DORA Article 28 third‑party register, because the trust plane is now somebody else's recovery procedure. Reasonable for a single‑region estate; ask the vendor to show you their evidence of the root's custody before you sign.
Restore the evidence store from backup instead of failing over. Cheaper, and it works for the application data. For evidence it means the restored bytes passed through a writable moment, so the verification step is no longer a nicety but the whole point. If you go this way, treat the verification report as the primary artefact and the restored bucket as untrusted until it exists.
Skip the DR test; keep the runbook. The most common choice and the one Article 12(2) specifically forbids. A runbook that has never been run against a segregated environment is a hypothesis.
From simple to standard, and what's next
The simple version is Velero for the namespaces, Argo CD for the manifests, and a hope that istiod comes back the same. The standard version is an offline root in its own jurisdiction, an intermediate in the backup, a six‑step order enforced by sync waves with deny‑all first, a pre‑registered federated credential for the recovery issuer, an Object‑Locked evidence replica, and a signed verification report that proves the ledger survived — produced on a schedule, on segregated systems, because Article 12(2) says so and because it is the only way you will believe it yourself.
Which raises the routine version of the same problem. You do not need a regional outage to replace a control plane. You do it every quarter, on purpose, and call it an upgrade. Next, Article 41: upgrading Istio without a control‑failure event — the traffic shift that is not a mutation.
DR isn't your app. It's your ability to still prove things afterwards.
Next in the series: Art. 41, upgrading Istio without a control‑failure event (Part G). Earlier: Art. 39 (multi‑cluster residency — the root's jurisdiction), Art. 28 (drift — record before you correct), Art. 27 (signed evidence), Art. 16 (default‑deny), Art. 15 (workload identity federation), Art. 14 (mTLS and SPIFFE).
Key skills & topics in this article
Compliance evidenceIstioDisaster recoveryGitOpsWorkload identitymTLSSupply-chain signingAuthorizationPolicy
Also on LinkedIn.