Article: Two Misconfigurations That Caused Spark OOM Failures on Kubernetes

Setting spark.kubernetes.local.dirs.tmpfs=true backs all scratch directories with node RAM, including shuffle spill, which can exhaust node memory within seconds during shuffle-heavy stages. A hard podAffinity rule that forces executor co-location concentrates shuffle-time memory pressure on a single node, which can trigger catastrophic out of memory (OOM) failures during shuffle-heavy stages. A one gibibyte (1Gi) scratch volume limit is insufficient for any shuffle-heavy job: size tmp-volume and workdir against your actual production shuffle data profile. Compound infrastructure misconfigurations often only surface under production-scale load and are easily misdiagnosed as Spark tuning problems like heap undersizing or data skew. Shortly after migrating several Spark batch pipelines from on-premises infrastructure to Azure Kubernetes Service (AKS), we began seeing repeated executor OOM failures in one of our larger jobs. The failures appeared during shuffle stages and initially looked like a typical Spark memory tuning problem. Executor memory was increased, executor counts were adjusted, and the job was restarted multiple times. None of these changes helped. The puzzling part was that the pipeline had been stable for years before the migration. The eventual root cause was not a Spark configuration issue at all. Instead, two infrastructure-level settings introduced during the migration interacted in an unexpected…