How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

Google Kubernetes Engine API connectivity failures blocked around 800 developers from deploying to production in January due to three interacting bugs across Private Service Connect, Envoy proxy behavior, and the Linux kernel. Packet captures revealed the control plane was responding to new TCP connection requests with stale ACK packets from previously closed connections.

Google Kubernetes Engine Failures

In January, a customer running multiple GKE clusters with Private Service Connect for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server. Analysis of packet captures showed the control plane was answering fresh TCP connection attempts by sending outdated ACK packets from connections that had already been terminated.

The root cause involved three separate bugs across different networking layers.

Akash Thakur, senior technical solutions engineer at Google Cloud, described the collaborative effort to isolate the issue:

“In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.”

Troubleshooting Complex Infrastructure Failures

The probe uncovered a trio of interconnected factors: packet sequence shuffling within Private Service Connect, Envoy’s half-close mechanism interfering with proper TCP connection termination, and a defect in the Linux kernel that caused certain reordered FIN packets to be dropped.

Thakur detailed a sophisticated debugging process that spanned Private Service Connect, virtual network interfaces, and the Linux TCP state machine, eventually uncovering a kernel flaw that caused sockets to hang forever in the FIN_WAIT_2 state.

Thakur explained the diagnostic challenge:

“Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — i/o timeout on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel’s TCP state machine, or Envoy’s connection management. Every layer had a plausible explanation, and every explanation was incomplete.”

Linux Kernel Patch Fixes GKE Timeouts

The issue was addressed through three coordinated fixes over several weeks. Connectivity was initially restored when Google reverted a Private Service Connect modification, followed by adjustments to proxy protocol management and the implementation of regression tests. Lastly, a Linux kernel update written by Eric Dumazet, who previously worked as a software engineer at Google, resolved the core TCP problem. Rohan Karne, managing director at Google Cloud, commented on the engineering effort:

“This is the unglamorous reality of mission-critical platform engineering—methodically ruling out the obvious suspects and digging into packet captures until you find the kernel root cause.”

Thakur highlighted how individually reasonable behaviors across networking, proxies, and operating systems can combine into complex infrastructure failures.

Examining packet captures across various network levels proved vital for pinpointing the fault, and linking network issues to infrastructure modifications helped clarify the relationship between the three bugs. Guangba Yu, postdoctoral researcher at the Chinese University of Hong Kong, highlighted the broader observational takeaway:

“For AIOps and chaos engineering, the transferable method is organizing simultaneous path observations, protocol state, and rollout events while distinguishing symptom recovery from state-machine correction.”

Yu also said that this anonymized case supplies no population failure rate and does not explain all API timeouts.

According to Justin Garrison, field CTO at Sidero, the kernel patch was incorporated into Linux 6.9.7 and subsequently backported to the 6.6 and 6.1 LTS releases, appearing for GKE users starting in version 1.33.5.