A. B. Khan

Upgrading Istio without a control‑failure event — the traffic shift that is not a mutation

The Mesh Series · Article 41 · Part G

Upgrades & patchingIstioAmbient meshCompliance evidenceDORAKubernetes operatorsMicrosoft Entra ID

Cover: Upgrading Istio without a control‑failure event

Istio 1.31.1 fixes GHSA‑qm8v‑g4f9‑qhjx: a BackendTLSPolicy that quietly reverts to plaintext when its CA reference fails to resolve. The control that encrypts was, under one specific failure, not encrypting. You read the advisory on a Tuesday. Your cluster is on 1.30.2. Your last control‑plane upgrade took a weekend, a change freeze and a rollback that nobody wants to talk about.

So the patch waits for the next quarterly window. In regulatory terms, you have just made a decision: for the next nine weeks, a known weakness in a control you cite in your DORA risk framework stays in production, because the process of removing it is scarier than the weakness.

One idea: an upgrade you are afraid to run is a vulnerability you have decided to keep

The fear is rational if an upgrade is a mutation — if istioctl upgrade replaces the one control plane your entire data plane depends on, in place, with a rollback path that is "reinstall the old one and hope." Every sidecar and every ztunnel talks to that one istiod. Replace it badly and every workload's configuration stops updating at once, every certificate renewal stalls, and the symptom is a slow, mesh‑wide decay with no single thing to point at.

The fear is irrational if an upgrade is a traffic shift. Istio's revision model lets two control planes run side by side: istiod-1-30-2 and istiod-1-31-1, each with its own revision label, each perfectly happy to coexist. A namespace chooses which one it talks to with a single label. Moving a namespace from the old revision to the new one is: change the label, restart the pods. Moving it back is: change the label back, restart the pods. Rollback is rollout in reverse, because the old control plane never left.

That reframing is the whole article. Once an upgrade is a label change per namespace, you can do it one namespace at a time, starting with the one that matters least, with a rollback that takes as long as a rollout restart. The quarterly window becomes a Tuesday‑afternoon job. The nine‑week weakness becomes a nine‑hour one.

Diagram: revision canary

Why it matters: patching is a statutory clock, not hygiene

DORA's Articles 6 to 9 make the ICT risk‑management framework a continuous obligation, and Article 9 — protection and prevention — is where vulnerability management lives. The RTS, 2024/1774, is where it bites: patching cadence tied to asset criticality, not to your release calendar. NIS2 Article 21(2)(e) puts vulnerability handling in the ten mandatory measures. RBI's framework expects patch status to be board‑visible. SAMA grades you on whether patching is a repeatable process, not on whether you happen to be current the week of the assessment.

Read those together and "we upgrade Istio quarterly" is not a policy; it is a confession that your patch latency for the mesh is up to ninety days regardless of CVSS.

Two scars make it concrete. CVE‑2026‑31837 (Article 23): a JWKS‑resolver failure causing the mesh to accept JWTs signed with a default key — CVSS 8.7, fixed in the 1.30 line. GHSA‑qm8v‑g4f9‑qhjx: encryption silently off, CVSS 6.8, fixed in 1.31.1. Both are failures of the control itself. Both are exactly what Article 38 would call a control‑failure event if they were exploited, with their own reporting clocks. And both sat in the open for anyone whose upgrade was scary.

Then there is the support window. Istio supports roughly three minors at a time; a release falls out of support about six weeks after N+2 ships. Stay two behind — a common "let it settle" policy — and you are not conservative. You are on an unsupported version the moment the next minor lands, and the next critical fix will not be backported to you. You will then have to do the big‑jump upgrade and the patch, under pressure, which is the one scenario where the fear was justified. Staying behind does not reduce upgrade risk. It concentrates it.

How it comes together: revisions, tags, and one label per namespace

Install by revision, never in place. istioctl install --revision=1-31-1 installs a second istiod deployment alongside the first. Nothing changes for any workload. The CRDs are upgraded — they are cluster‑scoped and shared — which is why the ambient order starts with them.

Address revisions through tags, not names. istioctl tag set prod-canary --revision 1-31-1 creates a stable name that namespaces point at. Namespaces never carry a version number; they carry istio.io/rev=prod-canary or istio.io/rev=prod-stable. Promotion is then one command — point prod-stable at the new revision — and no namespace label changes at all.

One label, one restart, one namespace. Set istio.io/rev=prod-canary on the least critical namespace, remove the legacy istio-injection=enabled label if it is there — it takes precedence and silently pins pods to the default revision — and kubectl rollout restart. Then look at the fact, not the intent: istioctl proxy-status tells you which istiod each proxy is actually connected to. The label is what you asked for; proxy-status is what happened, and it is the line that goes in the change record.

Respect the skew rule. The control plane may run one minor ahead of the data plane; the data plane must never be ahead of the control plane. So: istiod first, limbs after. In ambient the order is CRDs → istiod → istio‑cni → ztunnel → gateways → waypoints. ztunnel is a DaemonSet, so it rolls node by node with a sub‑second traffic blip per node — application pods do not restart, which is the ambient dividend, but the blip is real and belongs in the change notice. Waypoints last, and not deferred: the waypoint is where the Entra JWT path lives (Article 11), so "we'll do the L7 bit later" means "the identity control runs on the old version longer than anything else."

Promote, then retire — carefully. When every namespace has moved, point prod-stable at the new revision and uninstall the old one with istioctl uninstall --revision=1-30-2. One jagged edge: uninstalling a revision does not reinstate the gateways it owned. If your ingress gateway was installed under the old revision, reinstall it under the new one before the uninstall, or lose ingress on a Tuesday afternoon.

Through Git, not a terminal. The revision install, the tag, and every namespace label are manifests in the GitOps repository (Article 31). The merged PR is the change record; the proxy-status output attached to it is the evidence that the change took effect. A kubectl label typed by a human is an upgrade with no change record, which under DORA is an upgrade that did not happen correctly even if it worked.

Show the config: canary one namespace, then roll it back on purpose

The upgrade. Note the one line that removes the legacy label; it is the most common reason a canary "didn't take."

Code: canary upgrade

The rollback. It is the same shape as the rollout, which is the point. Run it in the DR test (Article 40) so the first time you need it is not the first time you have run it.

Code: rollback

Security · Audit · Integration

Security: the fix for a control‑level CVE ships to the least critical namespace within hours of the advisory, with the old control plane still running as the rollback; the exposure window for the whole estate is measured in namespace‑moves, not quarters.

Audit: every upgrade is a merged PR carrying a revision manifest, a tag change and a namespace label, with istioctl proxy-status output attached as the proof that intent became fact — DORA Article 9 and RTS 2024/1774's patch cadence demonstrated per asset, not asserted per quarter.

Integration: CRDs → istiod → cni → ztunnel → gateways → waypoints; the waypoint carries the Entra JWT path and is never the deferred step; gateways are re‑homed before a revision is retired; the DR test (Article 40) runs the rollback script so it is rehearsed, not hoped for.

Alternatives, and when you'd pick them

In‑place upgrade. istioctl upgrade against the default revision. Simplest, and fine for a dev cluster. It has no symmetric rollback — the old control plane is gone — so for anything under a change‑control regime it turns every upgrade into the weekend event this article is trying to abolish.

Stay N behind. "Let the release settle." Reasonable instinct, wrong arithmetic: with three supported minors and a six‑week overlap, N‑2 is unsupported almost immediately, and the next security fix does not come to you. Pick it only if you have a vendor contract that backports, and then it is not really N‑behind.

Managed mesh. The AKS Istio add‑on or OpenShift Service Mesh upgrades the control plane on the provider's schedule. You trade your patch latency for theirs, and the provider becomes a line in your DORA Article 28 third‑party register with concentration risk attached. Right for a team that cannot staff a mesh; ask for the provider's own patch‑latency evidence, since you will be citing it.

Freeze and pentest around it. Some institutions compensate for a stale mesh with a TLPT finding (Article 37) that says "no exploit path found." That is evidence a test did not find a path, not evidence the known weakness is gone. An assessor will read the advisory.

From simple to standard, and what's next

The simple version is one istiod, an in‑place upgrade in a quarterly window, and a rollback plan that says "reinstall." The standard version is two revisions side by side, tags that namespaces point at, one label per namespace moved through Git, proxy-status attached to the merged PR, limbs upgraded in order with the waypoint never deferred, gateways re‑homed before a revision retires, and a rollback script that the DR test runs on purpose. Nothing exotic; the revision model has been in Istio for years. What changes is that patching stops being brave.

Running two control planes side by side costs something, and so does keeping 180 days of signed evidence in‑country. Part G has been quietly spending money for three articles. Next, Article 42: what the mesh actually costs — two curves on two axes, and the one the CFO sees.

An upgrade you are afraid to run is a vulnerability you have decided to keep.


Next in the series: Art. 42, what the mesh actually costs (Part G closer). Earlier: Art. 40 (DR — rehearse the rollback), Art. 38 (a botched upgrade is a control‑failure event), Art. 31 (a merged PR is the change record), Art. 23 (CVE‑2026‑31837), Art. 11 (the waypoint carries the Entra path).

Key skills & topics in this article

Upgrades & patchingIstioAmbient meshCompliance evidenceDORAKubernetes operatorsMicrosoft Entra ID

  • #Istio
  • #Kubernetes
  • #ServiceMesh
  • #DORA
  • #NIS2
  • #PatchManagement
  • #Ambient
  • #GitOps
  • #ZeroTrust
  • #RegTech

Also on LinkedIn.