When the Kubernetes API went silent: A three-layer networking mystery

Every support engineer has a case that stays with them. The one that comes up at team lunches. The one where, even after it's resolved, you find yourself staring at a packet capture at midnight just to make sure you really understand what happened. This is that case. It started in January 2026 with a support ticket that read deceptively simply: pods are intermittently failing to reach the Kubernetes API server. The error was a familiar-looking timeout. The customer had already ruled out their own operators. What none of us knew yet was that we were looking at three separate bugs — in three different layers of the stack — all conspiring to produce the same symptom at the same time. By the time it was over, around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt. In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself. This is the story of how we got there. Part one: The alarm bell that wouldn't stop The customer ran a large multi-cluster GKE environment in Google Cloud's europe-west region. Their clusters used Private Service Connect — Google Cloud's mechanism for exposing the GKE control plane privately — and had a rich ecosystem of system-level workloads: Twistlock security scanners, External Secrets Operator for secrets management, Cilium for network policy, as well as several internal Kubernetes operators. The first errors appeared in one of its user acceptance testing (UAT) clusters in europe-west, and a few days later in a second cluster in the same region. They looked like this: Application logs — pods across multiple namespaces code_block <ListValue: [StructValue([('code', "TimeoutError: Connect call failed ('1.2.3.4', 443)\r\ndial tcp 1.2.3.4:443: i/o timeout\r\n\r\n# 1.2.3.4 is the cluster's internal VIP for the kube-apiserver,\r\n# routed through Private Service Connect"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f1ec2240c10>)])]> Initially the errors were intermittent — spikes in the logs, not a hard outage. The customer suspected their own code. Their team had recently deployed a custom operator built with the kopf framework, and it was the first thing they looked at. But the logs showed the errors had started eight days before the operator was ever deployed. When they shut it down completely, the timeouts kept coming. The issue persists even after completely scaling down the External Secrets Operator. The errors are appearing from customer workloads, and system-level pods we haven't touched. The growing list of suspects When a case like this arrives, the first instinct is to ask: what changed? The customer's timeline gave us several candidates, and we worked through them methodically. The custom operator (kopf): Ruled out immediately — errors predated deployment by a week and continued after shutdown. External Secrets Operator: Removed entirely on customer request. Timeouts continued in customer workloads that relied on Kubernetes control plane (KCP) connectivity, and cluster-internal system pods. Security agents: Uninstalled. Same result. We were now watching system-level Kubernetes components fail to reach their own control plane. conntrack table exhaustion: One theory was that too many half-open connections were filling the OS connection tracking table on the control plane nodes. We checked. The conntrack table was at roughly 50% capacity — of a default maximum of 1.3M entries. Not even close to exhaustion. Control plane overload: We scaled up the KCP VM size. kube-apiserver latencies looked healthy. No improvement on the timeout side. What made this case genuinely strange was the scope. We confirmed the timeouts were happening across five different clusters, affecting DaemonSets, Deployments, and system pods alike. External clients — developer laptops, GitHub Actions runners — were also hitting the same wall. This was not a broken workload. This was something wrong with the path between everything and the API server. The timeline so far Day 1: First I/O timeout errors appear (One of the UAT clusters in europe-west3. Intermittent, easy to dismiss.) Day 10: Errors become persistent (Frequency increases significantly. The issue is no longer dismissible.) Day 15: Case escalated — P0 bug filed (Multiple clusters confirmed affected. ~800 developers blocked. Production releases are frozen. Formal escalation begins.) Day 20: All obvious suspects ruled out (Custom operator, security agents, conntrack exhaustion, control plane resizing — all cleared.) Day 22: Packet captures collected (Captures taken simultaneously at the control-plane node NIC and from inside the

Excerpt shown under the publisher's RSS terms · 200 words · full text and figures are on the publisher's site