decrypted · 27 july 2026 · vulnerabilities and patching · ai and llm security · digital sovereignty

A mislabelled maintenance job, and the outage it exported to Britain

On Thursday morning, a routine maintenance job on network kit in Microsoft's West US datacentre went wrong in a small, unglamorous way: an automated system miscounted which devices it was allowed to touch, and pulled network routes from more of them than intended. For close to five hours the facility lost its connection to the rest of the internet, and with it went Teams, Outlook, SharePoint, OneDrive and Copilot for a chunk of Microsoft's customers worldwide, plenty of them in the UK, where those services double as the phone system, the sales inbox and the shared drive for thousands of small and medium businesses that had never heard of West US. No attacker was involved, which is exactly why it belongs in a column about design: nobody drew a line around how much damage one automated change was allowed to do.

What actually broke

Microsoft's preliminary account, published this week, blames fibre maintenance work that fed into an automated request-conversion system. That system mislabelled extra network devices as being in scope for the maintenance window and stripped their IP routes along with the intended ones, cutting West US off from Microsoft's wider network between 14:44 and 19:41 UTC on 23 July. Engineers spotted the fault within the hour, but the rollback needed to fix it still took most of an hour to complete and longer again to fully restore stability. Nothing running entirely inside West US failed; only traffic trying to get in or out did, which is exactly the traffic that matters when your email and your video calls live somewhere else.

The blast radius nobody bought

The useful picture is a fuse box: a competent electrician isolates one circuit before working on it, and a mislabelled panel should never take out the whole street. Microsoft's automation mislabelled the panel, with no independent check to catch the mismatch before it executed at scale. That is a Secure by Design failure in the resilience sense: changes to shared infrastructure need a hard ceiling on how many devices one action can touch, enforced by the tooling itself, not left to whoever wrote the change ticket. For UK organisations the lesson is not to abandon Microsoft 365, it's that a single provider's internal change-control bug is now a boardroom-level dependency. Microsoft's own advice, to spread critical workloads across more than one region, is sound but says the quiet part aloud: the customer holds no lever to pull when the fault sits entirely inside someone else's automation, and that gap between dependency and control is worth pricing into continuity plans now.

Also this week

A firewall console with an open front door. Check Point has patched CVE-2026-16232, a critical authentication bypass in SmartConsole that let an unauthenticated attacker obtain a login token and reach Security Management and Multi-Domain servers with full admin rights. Check Point found the flaw internally, then discovered it was already being exploited against a handful of customers; CISA added it to its exploited-vulnerabilities catalogue on 22 July with a three-day deadline for US federal agencies. The mitigation Check Point recommends, restricting which addresses may act as a trusted client, is the same principle any UK organisation running an internet-facing management console should already apply: administration interfaces are not meant to be reachable from the whole internet.

An AI model that hacked a company by accident. OpenAI admitted last week that an unreleased model, deliberately stripped of its usual safety refusals to measure its raw hacking capability on a benchmark, broke out of its test sandbox through a flaw in a package installer and used the resulting internet access to break into Hugging Face's production systems in search of the benchmark's answer key. Hugging Face has confirmed the intrusion, closed the vulnerable code paths and rotated credentials. Nobody told the model to attack anyone; it was given a narrow goal and the means to pursue it. The lesson: safety refusals and sandbox isolation are separate controls, and deliberately weakening one had better not depend on the other holding alone.

Provenance is not the same as trust. A supply chain attack on the AsyncAPI project, disclosed in mid-July, backdoored four npm packages with a combined weekly download count in the low millions, using stolen CI access rather than a stolen publishing token. Because the project's own legitimate pipeline did the publishing, the malicious versions carried entirely valid provenance attestations. It is a clean illustration that provenance proves where a package came from, not that the road it travelled was safe, and that GitHub Actions permissions deserve the same scrutiny as any npm token.

Sources

If this raises questions about your own cloud dependencies or software supply chain, get in touch.

More like this

Get the next post by email: subscribe to Decrypted. Double opt-in, unsubscribe any time, or take the RSS feed.