GitHub was down for 7h 47m. The retry storm was a separate failure.

The cascade

An Istio sidecar reached its limit for concurrent connections. The autoscaling policy that was supposed to catch this was configured against the capacity of the main service, not against the sidecar's own limits, so from the autoscaler's point of view nothing needed scaling. The sidecar saturated and stayed saturated.

Pressure moved outward from there. Four HAProxy nodes exhausted their flow limits, which degraded the gateway authentication path, which is why the blast radius reads like an unrelated list of products. Issues, Actions, SCIM and Copilot do not share much, but they all share the path that proves who you are.

None of this is exotic. It is the ordinary shape of a saturation cascade, and GitHub handled the recovery of it reasonably fast: they paused HAProxy on the affected nodes, and most services were healthy again by 16:36 UTC, about three hours in. Actions followed at 18:03.

The incident ran until 21:15.

An autoscaler on the wrong signal is worse than none

The temptation is to file this under "capacity problem" and move on. It is more specific than that, and the specificity is the useful part.

Autoscaling is a control loop, and a control loop is only as good as the thing it measures. If the constrained resource is the sidecar's connection concurrency and the policy is reading the main service's utilisation, the loop is closed around a variable that is not the one going critical. It reports healthy right up to the failure, and it does so correctly, by its own definition of health.

A system with no autoscaler fails loudly. A system autoscaling on the wrong signal fails while every dashboard stays green.

That is the part worth internalising, because the same mismatch is available in almost every stack. CPU and memory are easy to graph, and they are frequently not what runs out first. Connection pool slots, in-flight request concurrency, file descriptors, thread pools, queue depth, proxy flow limits and per-dependency quotas all fail earlier and none of them appear on a default dashboard. Sidecars make it worse, because the sidecar is a second process with its own ceilings sitting in the request path of a service that reports its own health independently.

The second failure

Here is what makes this incident worth reading rather than skimming. While the infrastructure was recovering, delayed responses from a single internal endpoint tripped a latent retry bug in VS Code. Traffic to the Copilot Token Service went from a normal 7,000–9,000 requests per second to somewhere between 70,000 and 100,000. Roughly ten times the load, arriving at a service that was already unwell.

The Copilot Token Service did not fully recover until 21:02 UTC.

Line the timestamps up and the shape of the incident changes. The original infrastructure failure ran from 13:28 to about 16:36, roughly three hours. The token service stayed down from 16:36 to 21:02, roughly four and a half. The aftermath lasted longer than the failure that caused it.

These were two different problems with two different fixes. The first was capacity, resolved by pausing HAProxy on the saturated nodes and correcting what the system was scaling against. The second was feedback, and you cannot fix feedback by adding capacity, because the clients will consume whatever you add and ask again.

Retries are a load generator

Retries get designed as a resilience feature, and against an isolated transient failure that is exactly what they are. Against a dependency that is already saturated, they are the opposite: every retry is another unit of load applied to the thing that is failing because of load. Slow responses produce more retries, more retries produce slower responses, and the loop sustains itself with no further help from the original fault.

The arithmetic is unforgiving. Ten thousand requests fail, every client retries twice, and the dependency does not see ten thousand requests. It sees something closer to thirty thousand attempts, arriving in a tighter window than the original traffic because failures are fast and everyone's backoff timer started at the same instant. Client retries also synchronise on their own, which is the entire reason jitter exists.

So the usual retry checklist is necessary and not sufficient. Max attempts, timeout, exponential backoff and jitter all bound a single client's behaviour. They say nothing about the aggregate. The questions that actually matter are about the population:

• What is the worst-case load this policy can generate across every client at once, and can the dependency survive that number?
• Can the callers synchronise? Anything that fails all clients simultaneously will restart all their backoff timers simultaneously.
• Is retry traffic distinguishable from first-attempt traffic at the server? If it is not, you cannot shed it selectively, and you cannot even see the storm in your own metrics.
• Can you shed or isolate this dependency deliberately, without a deploy, while it is on fire?

The goal is not to retry until it works. The goal is to fail in a way that leaves the dependency enough headroom to come back.

The mitigation is the most interesting detail

GitHub's fix for the storm was to reduce retry logic in the gateway and temporarily block some token requests outright. Read that again as an engineering decision: to recover the service, they had to stop serving their own clients on purpose. Circuit breaking, applied by hand, under pressure.

The client here was VS Code, which GitHub ships. That is close to the best case, and it still took hours, because a fix in a client that is already installed on millions of machines does not reach those machines on incident timescales. Nobody restarts their editor because a status page asked them to.

You cannot patch your clients during an incident. Retry policy is a design-time decision that you live with, not a lever you reach for when things break.

Which means the server side has to assume its clients will misbehave, and has to keep the ability to refuse them. Rate limits, load shedding and circuit breakers are not there to punish callers; they are the only controls that still work when the callers are the problem and you cannot change the callers.

Recovery is its own workload

Most capacity planning models two states: normal and failed. This incident is a clean argument for a third one. Recovery has a load profile that resembles neither, and it can be the largest of the three.

Caches come back empty and every read goes to the origin. Every client that was disconnected reconnects, and they do it together. Queues that absorbed the outage drain all at once. Workers restart and re-register. Health checks intensify. And the clients that were retrying start succeeding, which for a while means more work completing per second than the system ever handles at steady state.

A system that survives the original failure can still fall over on the way back up, and it will do so at the moment the incident channel has started to relax. If recovery load has never been measured, it is not a known quantity, it is a bet.

What I take from it

GitHub's own follow-ups are the obvious ones and they are correct: fix the autoscaling policies, audit the Istio limits, review retry and backoff behaviour in both gateway and clients, address the VS Code amplification, improve load balancer capacity monitoring and regional failover.

The transferable lessons are smaller and cheaper than that, and none of them require operating at GitHub's scale to be worth doing:

• Scale on the resource that saturates first, not the one that is easiest to graph. Write it down per service, and check that the alert and the scaling policy agree with the note.
• Give every sidecar and proxy in the request path its own limits and its own alerts. A healthy service behind an unhealthy sidecar still reads as healthy.
• Budget retries in aggregate. Size the policy against total client population, not per-client politeness, and confirm the dependency survives that number.
• Tag retry traffic so it is separable at the server. You want to be able to see and drop it independently.
• Keep a shed switch you can throw without a deploy. If the only way to reduce load is to ship code, you do not have the control.
• Load test the recovery, not only the failure. Cold caches, reconnect storms and queue drain are a distinct workload and should be measured as one.

The headline conclusion from an outage like this is that GitHub should have scaled better. The more useful one is that a distributed system can fail because you were watching the wrong bottleneck, and can then stay failed because the mechanisms designed to make it resilient are generating the load that is keeping it down.

The first problem is capacity. The second is feedback. The second is harder to spot, because by the time it dominates, the original fault is usually fixed and the graphs are already pointing the right way.

It is a good argument for reading incident reports past the root cause. The valuable material is rarely what broke. It is what happened after the engineers started fixing it.